[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:08 dist-test-slave-0lgr krb5kdc[11344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:08 dist-test-slave-0lgr krb5kdc[11344](info): set up 2 sockets
Sep 24 19:12:08 dist-test-slave-0lgr krb5kdc[11344](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 19:12:09.999229 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:10 dist-test-slave-0lgr krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:10.186959 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38785
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:38785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:10.527496 11357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:10.527837 11357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:10.528014 11357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:10.536224 11357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:10.536357 11357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:10.536460 11357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:10.536550 11357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:10.536618 11357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:10.548388 11357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:38785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:10.549988 11357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:10.551568 11357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:10.566061 11363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:10.566541 11365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:10.566545 11362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:10.567812 11357 server_base.cc:1034] running on GCE node
I20260924 19:12:10.568763 11357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:10.570700 11357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:10.571977 11357 hybrid_clock.cc:648] HybridClock initialized: now 1790277130571845 us; error 145 us; skew 500 ppm
I20260924 19:12:10.572434 11357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:10 dist-test-slave-0lgr krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277130, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:10.581414 11357 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:10.583204 11357 webserver.cc:460] Webserver started at http://127.11.18.126:44281/ using document root <none> and password file <none>
I20260924 19:12:10.583936 11357 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:10.584041 11357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:10.584309 11357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:10.587524 11357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "8038251802d8474d81f9d70b7d1e04dd"
format_stamp: "Formatted at 2026-09-24 19:12:10 on dist-test-slave-0lgr"
server_key: "a2e53bd37d7bdd51fb44b78a497703f9"
server_key_iv: "d8cae7a7ef6433be2be097ed6b0686f8"
server_key_version: "encryptionkey@0"
I20260924 19:12:10.588299 11357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "8038251802d8474d81f9d70b7d1e04dd"
format_stamp: "Formatted at 2026-09-24 19:12:10 on dist-test-slave-0lgr"
server_key: "a2e53bd37d7bdd51fb44b78a497703f9"
server_key_iv: "d8cae7a7ef6433be2be097ed6b0686f8"
server_key_version: "encryptionkey@0"
I20260924 19:12:10.594619 11357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:12:10.599208 11372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:10.601086 11357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:12:10.601372 11357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "8038251802d8474d81f9d70b7d1e04dd"
format_stamp: "Formatted at 2026-09-24 19:12:10 on dist-test-slave-0lgr"
server_key: "a2e53bd37d7bdd51fb44b78a497703f9"
server_key_iv: "d8cae7a7ef6433be2be097ed6b0686f8"
server_key_version: "encryptionkey@0"
I20260924 19:12:10.601612 11357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:10.619395 11357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:10.624634 11357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:10.624975 11357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:10.646164 11357 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:38785
I20260924 19:12:10.646166 11423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:38785 every 8 connection(s)
I20260924 19:12:10.647760 11357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:10.652097 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:10.655687 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 11357
I20260924 19:12:10.656435 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:10.658583 11337 external_mini_cluster.cc:1428] Setting key 88cf11f95751f77bd16e9da0635d29d3
I20260924 19:12:10.664054 11424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd: Bootstrap starting.
I20260924 19:12:10.667969 11424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:10.669600 11424 log.cc:824] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:10.672775 11424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd: No bootstrap required, opened a new log
I20260924 19:12:10.681806 11424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8038251802d8474d81f9d70b7d1e04dd" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38785 } }
I20260924 19:12:10.682266 11424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:10.682385 11424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8038251802d8474d81f9d70b7d1e04dd, State: Initialized, Role: FOLLOWER
I20260924 19:12:10.683010 11424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8038251802d8474d81f9d70b7d1e04dd" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38785 } }
I20260924 19:12:10.683269 11424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:10.683414 11424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:10.683585 11424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:10.686008 11424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8038251802d8474d81f9d70b7d1e04dd" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38785 } }
I20260924 19:12:10.686508 11424 leader_election.cc:302] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8038251802d8474d81f9d70b7d1e04dd; no voters:
I20260924 19:12:10.687202 11424 leader_election.cc:288] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:10.687322 11429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:10.689868 11429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [term 1 LEADER]: Becoming Leader. State: Replica: 8038251802d8474d81f9d70b7d1e04dd, State: Running, Role: LEADER
I20260924 19:12:10.689980 11424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:10.690467 11429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8038251802d8474d81f9d70b7d1e04dd" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38785 } }
Sep 24 19:12:10 dist-test-slave-0lgr krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:10.695037 11430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8038251802d8474d81f9d70b7d1e04dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8038251802d8474d81f9d70b7d1e04dd" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38785 } } }
I20260924 19:12:10.695518 11430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:10.696209 11431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8038251802d8474d81f9d70b7d1e04dd. Latest consensus state: current_term: 1 leader_uuid: "8038251802d8474d81f9d70b7d1e04dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8038251802d8474d81f9d70b7d1e04dd" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38785 } } }
I20260924 19:12:10.696700 11431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:10.700579 11438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:10.708832 11438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:10.709236 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:10.663331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.11.18.126:38785)
0924 19:12:10.664197 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:10.664215 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:10.665621 (+ 1406us) server_negotiation.cc:407] Connection header received
0924 19:12:10.675383 (+ 9762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:10.675409 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:10.675906 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:10.676455 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:10.680393 (+ 3938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:10.681953 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:10.684362 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:10.685164 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:10.685425 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:10.694106 (+ 8681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:10.694125 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:10.694163 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:10.694246 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:10.701202 (+ 6956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:10.702211 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:10.702220 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:10.702228 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:10.702421 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:10.702935 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:10.702940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:10.702944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:10.703531 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:10.706843 (+ 3312us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:10.707139 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 19:12:10.707621 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":400,"threads_started":1}
I20260924 19:12:10.721848 11438 catalog_manager.cc:1347] Generated new cluster ID: 9c078943520c4676b28c56d49fbf160d
I20260924 19:12:10.721989 11438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:10.737126 11438 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:10.738682 11438 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:10.748590 11438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8038251802d8474d81f9d70b7d1e04dd: Generated new TSK 0
I20260924 19:12:10.749683 11438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:12:10.798260 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38785
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:11.166435 11450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:11.166767 11450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:11.166877 11450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:11.175573 11450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:11.175709 11450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:11.175837 11450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:11.187180 11450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:11.188731 11450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:11.190608 11450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:11.205641 11455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:11.205746 11456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:11.206389 11458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:11.207074 11450 server_base.cc:1034] running on GCE node
I20260924 19:12:11.207870 11450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:11.208817 11450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:11.210018 11450 hybrid_clock.cc:648] HybridClock initialized: now 1790277131209970 us; error 51 us; skew 500 ppm
I20260924 19:12:11.210398 11450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:11 dist-test-slave-0lgr krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277131, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:11.213801 11450 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:12:11.215953 11450 webserver.cc:460] Webserver started at http://127.11.18.65:41231/ using document root <none> and password file <none>
I20260924 19:12:11.216663 11450 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:11.216766 11450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:11.217042 11450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:11.220167 11450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "d029c38cc39540489af7009d50428079"
format_stamp: "Formatted at 2026-09-24 19:12:11 on dist-test-slave-0lgr"
server_key: "463c0cda4e35b990d71775c59ba01530"
server_key_iv: "39d77d4a274951e73f6faa3fbcad9edf"
server_key_version: "encryptionkey@0"
I20260924 19:12:11.221211 11450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "d029c38cc39540489af7009d50428079"
format_stamp: "Formatted at 2026-09-24 19:12:11 on dist-test-slave-0lgr"
server_key: "463c0cda4e35b990d71775c59ba01530"
server_key_iv: "39d77d4a274951e73f6faa3fbcad9edf"
server_key_version: "encryptionkey@0"
I20260924 19:12:11.227427 11450 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 19:12:11.231441 11465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:11.232930 11450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:12:11.233081 11450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "d029c38cc39540489af7009d50428079"
format_stamp: "Formatted at 2026-09-24 19:12:11 on dist-test-slave-0lgr"
server_key: "463c0cda4e35b990d71775c59ba01530"
server_key_iv: "39d77d4a274951e73f6faa3fbcad9edf"
server_key_version: "encryptionkey@0"
I20260924 19:12:11.233268 11450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:11.250814 11450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:11.257268 11450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:11.257681 11450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:11.259177 11450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:11.261552 11450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:11.261656 11450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:11.261790 11450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:11.261875 11450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:11.300297 11450 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:43379
I20260924 19:12:11.300472 11577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:43379 every 8 connection(s)
I20260924 19:12:11.302027 11450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:11.308559 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 11450
I20260924 19:12:11.308813 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:11.309655 11337 external_mini_cluster.cc:1428] Setting key 6c1626f0641f93bafd3d5fefb18a3f1a
Sep 24 19:12:11 dist-test-slave-0lgr krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277131, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:12:11.328233 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:11.305108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:37647 (local address 127.11.18.126:38785)
0924 19:12:11.305630 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:11.305634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:11.307230 (+ 1596us) server_negotiation.cc:407] Connection header received
0924 19:12:11.309887 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:11.309896 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:11.310010 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:11.310125 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:11.312048 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:11.312830 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:11.315661 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:11.316152 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:11.316279 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:11.320837 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:11.320843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:11.320847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:11.320888 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:11.324533 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:11.325575 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:11.325580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:11.325582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:11.325716 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:11.326226 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:11.326230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:11.326233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:11.326568 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:11.326748 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:11.327576 (+ 828us) server_negotiation.cc:299] Negotiation successful
0924 19:12:11.327841 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":302,"threads_started":1}
I20260924 19:12:11.329309 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:11.306164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38785 (local address 127.11.18.65:37647)
0924 19:12:11.307045 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:11.307095 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:11.309402 (+ 2307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:11.310442 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:11.310452 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:11.311129 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:11.311836 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:11.311857 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:11.313034 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:11.313043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:11.315455 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:11.315477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:11.316434 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:11.316445 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:11.316817 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:11.317624 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:11.317654 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:11.320528 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:11.324750 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:11.324762 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:11.324829 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:11.325409 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:11.325859 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:11.325865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:11.325869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:11.326082 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:11.326719 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:11.326728 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:11.327161 (+ 433us) client_negotiation.cc:769] Sending connection context
0924 19:12:11.327462 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 19:12:11.328074 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
I20260924 19:12:11.331775 11578 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38785
I20260924 19:12:11.332152 11578 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:11.333109 11578 heartbeater.cc:511] Master 127.11.18.126:38785 requested a full tablet report, sending...
I20260924 19:12:11.335853 11389 ts_manager.cc:194] Registered new tserver with Master: d029c38cc39540489af7009d50428079 (127.11.18.65:43379)
I20260924 19:12:11.337709 11389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:37647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:12:11.357409 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38785
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:11.717238 11584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:11.717625 11584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:11.717813 11584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:11.725585 11584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:11.725739 11584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:11.725914 11584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:11.736320 11584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:11.737779 11584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:11.739178 11584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:11.752856 11589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:11.752908 11592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:11.753034 11590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:11.754703 11584 server_base.cc:1034] running on GCE node
I20260924 19:12:11.755335 11584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:11.756395 11584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:11.757624 11584 hybrid_clock.cc:648] HybridClock initialized: now 1790277131757611 us; error 86 us; skew 500 ppm
I20260924 19:12:11.757961 11584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:11 dist-test-slave-0lgr krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277131, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:11.761296 11584 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:12:11.762889 11584 webserver.cc:460] Webserver started at http://127.11.18.66:43411/ using document root <none> and password file <none>
I20260924 19:12:11.763598 11584 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:11.763744 11584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:11.764058 11584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:11.767052 11584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "6ea783f9f64f47d38ed322b30d590678"
format_stamp: "Formatted at 2026-09-24 19:12:11 on dist-test-slave-0lgr"
server_key: "99338e15f9aa5f6305abfba064dd7ac6"
server_key_iv: "223495a532b1d3f7fe9d6f216c9cb38d"
server_key_version: "encryptionkey@0"
I20260924 19:12:11.767848 11584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "6ea783f9f64f47d38ed322b30d590678"
format_stamp: "Formatted at 2026-09-24 19:12:11 on dist-test-slave-0lgr"
server_key: "99338e15f9aa5f6305abfba064dd7ac6"
server_key_iv: "223495a532b1d3f7fe9d6f216c9cb38d"
server_key_version: "encryptionkey@0"
I20260924 19:12:11.773231 11584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:12:11.777603 11599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:11.779207 11584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 19:12:11.779461 11584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "6ea783f9f64f47d38ed322b30d590678"
format_stamp: "Formatted at 2026-09-24 19:12:11 on dist-test-slave-0lgr"
server_key: "99338e15f9aa5f6305abfba064dd7ac6"
server_key_iv: "223495a532b1d3f7fe9d6f216c9cb38d"
server_key_version: "encryptionkey@0"
I20260924 19:12:11.779699 11584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:11.803478 11584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:11.808856 11584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:11.809166 11584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:11.810232 11584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:11.811700 11584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:11.811771 11584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:11.811937 11584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:11.811988 11584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:11.849793 11584 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:41739
I20260924 19:12:11.849830 11711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:41739 every 8 connection(s)
I20260924 19:12:11.851186 11584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:11.856726 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 11584
I20260924 19:12:11.856938 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:11.857542 11337 external_mini_cluster.cc:1428] Setting key b319a43fd38075492f81d18a4ef750ec
Sep 24 19:12:11 dist-test-slave-0lgr krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277131, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:12:11.878098 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:11.854182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:56757 (local address 127.11.18.126:38785)
0924 19:12:11.854778 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:11.854785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:11.855520 (+ 735us) server_negotiation.cc:407] Connection header received
0924 19:12:11.858354 (+ 2834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:11.858361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:11.858461 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:11.858572 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:11.860623 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:11.861432 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:11.864818 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:11.865620 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:11.865761 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:11.870345 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:11.870355 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:11.870364 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:11.870416 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:11.874274 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:11.875315 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:11.875324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:11.875331 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:11.875497 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:11.876114 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:11.876119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:11.876121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:11.876517 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:11.876642 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:11.877568 (+ 926us) server_negotiation.cc:299] Negotiation successful
0924 19:12:11.877793 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:12:11.879230 11715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:11.854575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38785 (local address 127.11.18.66:56757)
0924 19:12:11.855341 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:11.855390 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:11.857909 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:11.858893 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:11.858903 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:11.859562 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:11.860419 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:11.860447 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:11.861604 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:11.861613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:11.864591 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:11.864608 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:11.865873 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:11.865879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:11.866111 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:11.866871 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:11.866900 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:11.870036 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:11.874631 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:11.874640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:11.874656 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:11.875163 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:11.875786 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:11.875790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:11.875792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:11.875988 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:11.876650 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:11.876657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:11.877008 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 19:12:11.877401 (+ 393us) client_negotiation.cc:240] Negotiation successful
0924 19:12:11.877856 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":311,"threads_started":1}
I20260924 19:12:11.880775 11712 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38785
I20260924 19:12:11.881104 11712 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:11.881907 11712 heartbeater.cc:511] Master 127.11.18.126:38785 requested a full tablet report, sending...
I20260924 19:12:11.883765 11389 ts_manager.cc:194] Registered new tserver with Master: 6ea783f9f64f47d38ed322b30d590678 (127.11.18.66:41739)
I20260924 19:12:11.884786 11389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:56757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:12:11.910706 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38785
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:12.261003 11718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:12.261309 11718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:12.261484 11718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:12.268889 11718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:12.269006 11718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:12.269129 11718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:12.283294 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:12.284806 11718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:12.286350 11718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:12.300953 11724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:12.300979 11726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:12.301738 11718 server_base.cc:1034] running on GCE node
W20260924 19:12:12.305666 11723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:12.306510 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:12.307669 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:12.308895 11718 hybrid_clock.cc:648] HybridClock initialized: now 1790277132308848 us; error 40 us; skew 500 ppm
I20260924 19:12:12.309393 11718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:12 dist-test-slave-0lgr krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277132, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:12.312479 11718 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:12:12.314357 11718 webserver.cc:460] Webserver started at http://127.11.18.67:37807/ using document root <none> and password file <none>
I20260924 19:12:12.314996 11718 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:12.315090 11718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:12.315353 11718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:12.318269 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "d8dfd49d088e437383d3c13922081b0c"
format_stamp: "Formatted at 2026-09-24 19:12:12 on dist-test-slave-0lgr"
server_key: "9a83d5680437f018f02f75a596c2ed62"
server_key_iv: "509b8e763fe1d3cc1aff49b6f0974d5f"
server_key_version: "encryptionkey@0"
I20260924 19:12:12.318964 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "d8dfd49d088e437383d3c13922081b0c"
format_stamp: "Formatted at 2026-09-24 19:12:12 on dist-test-slave-0lgr"
server_key: "9a83d5680437f018f02f75a596c2ed62"
server_key_iv: "509b8e763fe1d3cc1aff49b6f0974d5f"
server_key_version: "encryptionkey@0"
I20260924 19:12:12.324664 11718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:12:12.328342 11733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:12.329959 11718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:12.330138 11718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "d8dfd49d088e437383d3c13922081b0c"
format_stamp: "Formatted at 2026-09-24 19:12:12 on dist-test-slave-0lgr"
server_key: "9a83d5680437f018f02f75a596c2ed62"
server_key_iv: "509b8e763fe1d3cc1aff49b6f0974d5f"
server_key_version: "encryptionkey@0"
I20260924 19:12:12.330339 11718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:12.341578 11718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:12.341641 11578 heartbeater.cc:503] Master 127.11.18.126:38785 was elected leader, sending a full tablet report...
I20260924 19:12:12.347046 11718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:12.347354 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:12.348416 11718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:12.349777 11718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:12.349850 11718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:12.349987 11718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:12.350035 11718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:12.385493 11718 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:39969
I20260924 19:12:12.385586 11845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:39969 every 8 connection(s)
I20260924 19:12:12.386809 11718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:12.388960 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 11718
I20260924 19:12:12.389158 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:12.389770 11337 external_mini_cluster.cc:1428] Setting key b0a9ff422e1dda32da055f8fbce8c748
Sep 24 19:12:12 dist-test-slave-0lgr krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277132, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:12.409451 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:12.389567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:38789 (local address 127.11.18.126:38785)
0924 19:12:12.390073 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:12.390080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:12.391282 (+ 1202us) server_negotiation.cc:407] Connection header received
0924 19:12:12.393581 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:12.393584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:12.393661 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:12.393750 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:12.395300 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.396278 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.398077 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.398503 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.398605 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:12.402434 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:12.402439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:12.402444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:12.402490 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:12.406102 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.407119 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.407123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.407125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.407245 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.407731 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.407735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.407737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.408011 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:12.408139 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:12.408869 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 19:12:12.409034 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":242,"threads_started":1}
I20260924 19:12:12.410521 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:12.390195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38785 (local address 127.11.18.67:38789)
0924 19:12:12.391104 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:12.391155 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:12.393235 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:12.394046 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:12.394054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:12.394571 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:12.395116 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:12.395133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.396439 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.396451 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:12.397948 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:12.397964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.398715 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.398721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:12.398926 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:12.399735 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:12.399759 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:12.402200 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:12.406272 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:12.406280 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:12.406294 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:12.406979 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:12.407345 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:12.407348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:12.407350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:12.407534 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:12.408148 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:12.408156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:12.408512 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 19:12:12.408750 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 19:12:12.409209 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":349,"threads_started":1}
I20260924 19:12:12.412098 11846 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38785
I20260924 19:12:12.412443 11846 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:12.413163 11846 heartbeater.cc:511] Master 127.11.18.126:38785 requested a full tablet report, sending...
I20260924 19:12:12.414755 11389 ts_manager.cc:194] Registered new tserver with Master: d8dfd49d088e437383d3c13922081b0c (127.11.18.67:39969)
I20260924 19:12:12.415630 11389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:38789
I20260924 19:12:12.419624 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:12:12 dist-test-slave-0lgr krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:12:12 dist-test-slave-0lgr krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:12.464514 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:12.447820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.11.18.65:43379)
0924 19:12:12.448492 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:12.448499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:12.448527 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:12:12.448712 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:12.448719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:12.448954 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:12.449090 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:12.450676 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.451679 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.453004 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.453685 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.453793 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:12.458280 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:12.458302 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:12.458322 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:12.458375 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:12.461394 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.462050 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.462057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.462060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.462188 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.462607 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.462611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.462614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.462977 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:12.463209 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:12.463584 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 19:12:12.463810 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":342,"threads_started":1}
W20260924 19:12:12.466055 11493 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53844
Sep 24 19:12:12 dist-test-slave-0lgr krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:12:12 dist-test-slave-0lgr krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:12.511121 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:12.495730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.11.18.65:43379)
0924 19:12:12.495962 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:12.495968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:12.496269 (+ 301us) server_negotiation.cc:407] Connection header received
0924 19:12:12.496508 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:12.496520 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:12.496589 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:12.496705 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:12.498094 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.498846 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.500173 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.500602 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.500742 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:12.505064 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:12.505069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:12.505072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:12.505112 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:12.508408 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.509183 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.509189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.509192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.509379 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.509737 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.509744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.509747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.510012 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:12.510252 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:12.510550 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 19:12:12.510788 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:12:12.513154 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 11450
I20260924 19:12:12.527366 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:12.514212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.11.18.65:43379)
0924 19:12:12.514386 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:12.514390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:12.514825 (+ 435us) server_negotiation.cc:407] Connection header received
0924 19:12:12.515017 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:12.515022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:12.515088 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:12.515168 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:12.516444 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.517156 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.518520 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.519351 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.519495 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:12.522042 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:12.522047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:12.522050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:12.522082 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:12.524740 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.525446 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.525453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.525457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.525600 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.525967 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.525974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.525977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.526239 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:12.526478 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:12.526769 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 19:12:12.526985 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:12:12.528119 11573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:12.615249 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 11450
I20260924 19:12:12.633234 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 11584
Sep 24 19:12:12 dist-test-slave-0lgr krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:12:12.651449 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:12.634308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.11.18.66:41739)
0924 19:12:12.634843 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:12.634850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:12.634875 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:12:12.634934 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:12.634940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:12.635155 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:12.635366 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:12.636690 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.637873 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.639357 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.639957 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.640072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:12.644630 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:12.644644 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:12.644664 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:12.644709 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:12.648218 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.649009 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.649018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.649020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.649164 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.649757 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.649764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.649772 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.650121 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:12.650450 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:12.650802 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 19:12:12.651075 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":305,"threads_started":1}
I20260924 19:12:12.652741 11707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:12.738031 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 11584
I20260924 19:12:12.755136 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 11718
Sep 24 19:12:12 dist-test-slave-0lgr krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:12:12.771929 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:12.756312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.11.18.67:39969)
0924 19:12:12.756911 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:12.756918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:12.756946 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:12:12.757003 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:12.757009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:12.757215 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:12.757505 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:12.758744 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.759820 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.761107 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:12.761783 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:12.761933 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:12.766269 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:12.766286 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:12.766301 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:12.766336 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:12.768949 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.769832 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.769848 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.769850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.769968 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:12.770337 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:12.770344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:12.770347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:12.770687 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:12.771015 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:12.771308 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 19:12:12.771576 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":294,"threads_started":1}
I20260924 19:12:12.773447 11841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:12.859759 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 11718
I20260924 19:12:12.875958 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 11357
I20260924 19:12:12.877441 11419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:12.982648 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 11357
2026-09-24T19:12:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4124 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:13 dist-test-slave-0lgr krb5kdc[11868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:13 dist-test-slave-0lgr krb5kdc[11868](info): set up 2 sockets
Sep 24 19:12:13 dist-test-slave-0lgr krb5kdc[11868](info): commencing operation
krb5kdc: starting...
W20260924 19:12:14.134147 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:14 dist-test-slave-0lgr krb5kdc[11868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:14.330229 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45333
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:14.657811 11881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:14.658192 11881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:14.658353 11881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:14.665882 11881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:14.666039 11881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:14.666146 11881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:14.666255 11881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:14.666332 11881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:14.676456 11881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:14.677884 11881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:14.679323 11881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:14.691962 11886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:14.691874 11889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:14.691926 11887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:14.692809 11881 server_base.cc:1034] running on GCE node
I20260924 19:12:14.693763 11881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:14.695390 11881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:14.696702 11881 hybrid_clock.cc:648] HybridClock initialized: now 1790277134696618 us; error 81 us; skew 500 ppm
I20260924 19:12:14.697093 11881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:14 dist-test-slave-0lgr krb5kdc[11868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277134, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:14.700557 11881 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:14.702360 11881 webserver.cc:460] Webserver started at http://127.11.18.126:33211/ using document root <none> and password file <none>
I20260924 19:12:14.703097 11881 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:14.703253 11881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:14.703595 11881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:14.706828 11881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "b5c7186eb9fb4b99baea48be7600628d"
format_stamp: "Formatted at 2026-09-24 19:12:14 on dist-test-slave-0lgr"
server_key: "302232e23c79c4b4b7de48f232b63136"
server_key_iv: "a98e2f5fc6adb838b5becba8079d7a93"
server_key_version: "encryptionkey@0"
I20260924 19:12:14.707600 11881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "b5c7186eb9fb4b99baea48be7600628d"
format_stamp: "Formatted at 2026-09-24 19:12:14 on dist-test-slave-0lgr"
server_key: "302232e23c79c4b4b7de48f232b63136"
server_key_iv: "a98e2f5fc6adb838b5becba8079d7a93"
server_key_version: "encryptionkey@0"
I20260924 19:12:14.713894 11881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 19:12:14.717801 11896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:14.719391 11881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:12:14.719606 11881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "b5c7186eb9fb4b99baea48be7600628d"
format_stamp: "Formatted at 2026-09-24 19:12:14 on dist-test-slave-0lgr"
server_key: "302232e23c79c4b4b7de48f232b63136"
server_key_iv: "a98e2f5fc6adb838b5becba8079d7a93"
server_key_version: "encryptionkey@0"
I20260924 19:12:14.719846 11881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:14.742324 11881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:14.747015 11881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:14.747323 11881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:14.767833 11881 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:45333
I20260924 19:12:14.767875 11947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:45333 every 8 connection(s)
I20260924 19:12:14.769289 11881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:14.773351 11948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:14.779024 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 11881
I20260924 19:12:14.779289 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:14.779959 11337 external_mini_cluster.cc:1428] Setting key 1a0818c81653ee9e9df462d8189c1b1c
I20260924 19:12:14.785597 11948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d: Bootstrap starting.
I20260924 19:12:14.788928 11948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:14.790271 11948 log.cc:824] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:14.793190 11948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d: No bootstrap required, opened a new log
Sep 24 19:12:14 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:14.801657 11948 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c7186eb9fb4b99baea48be7600628d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45333 } }
I20260924 19:12:14.802074 11948 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:14.802211 11948 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c7186eb9fb4b99baea48be7600628d, State: Initialized, Role: FOLLOWER
I20260924 19:12:14.802805 11948 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c7186eb9fb4b99baea48be7600628d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45333 } }
I20260924 19:12:14.803063 11948 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:14.803207 11948 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:14.803395 11948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:14.803722 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:14.782089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.11.18.126:45333)
0924 19:12:14.782973 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:14.782988 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:14.783034 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:12:14.784919 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:14.784944 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:14.785468 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:14.785921 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:14.787505 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:14.788807 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:14.790224 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:14.790989 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:14.791234 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:14.795592 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:14.795601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:14.795617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:14.795670 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:14.799156 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:14.799978 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:14.799985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:14.799992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:14.800156 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:14.800542 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:14.800546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:14.800548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:14.801084 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:14.801409 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:14.801899 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 19:12:14.802524 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":410,"threads_started":1}
I20260924 19:12:14.805994 11948 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c7186eb9fb4b99baea48be7600628d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45333 } }
I20260924 19:12:14.806515 11948 leader_election.cc:302] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5c7186eb9fb4b99baea48be7600628d; no voters:
I20260924 19:12:14.807050 11948 leader_election.cc:288] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:14.807286 11953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:14.808043 11953 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [term 1 LEADER]: Becoming Leader. State: Replica: b5c7186eb9fb4b99baea48be7600628d, State: Running, Role: LEADER
I20260924 19:12:14.808583 11953 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c7186eb9fb4b99baea48be7600628d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45333 } }
I20260924 19:12:14.809206 11948 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:14.811568 11955 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5c7186eb9fb4b99baea48be7600628d. Latest consensus state: current_term: 1 leader_uuid: "b5c7186eb9fb4b99baea48be7600628d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c7186eb9fb4b99baea48be7600628d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45333 } } }
I20260924 19:12:14.811594 11954 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5c7186eb9fb4b99baea48be7600628d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c7186eb9fb4b99baea48be7600628d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45333 } } }
I20260924 19:12:14.812024 11955 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:14.812024 11954 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:14.814226 11960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:14.818627 11960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:14.828461 11960 catalog_manager.cc:1347] Generated new cluster ID: dd1a0ae5e74e49e4a4d7e7613586dd3c
I20260924 19:12:14.828600 11960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:14.838084 11960 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:14.839830 11960 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:14.845376 11960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5c7186eb9fb4b99baea48be7600628d: Generated new TSK 0
I20260924 19:12:14.846442 11960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:12:14.889622 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45333
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:15.232811 11974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:15.233119 11974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:15.233269 11974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:15.240778 11974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:15.240904 11974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:15.241096 11974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:15.251425 11974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:15.252874 11974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:15.254258 11974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:15.268539 11979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:15.268560 11982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:15.268580 11980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:15.269707 11974 server_base.cc:1034] running on GCE node
I20260924 19:12:15.270373 11974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:15.271569 11974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:15.272812 11974 hybrid_clock.cc:648] HybridClock initialized: now 1790277135272760 us; error 43 us; skew 500 ppm
I20260924 19:12:15.273135 11974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:15 dist-test-slave-0lgr krb5kdc[11868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277135, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:15.282621 11974 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:12:15.284305 11974 webserver.cc:460] Webserver started at http://127.11.18.65:40379/ using document root <none> and password file <none>
I20260924 19:12:15.285056 11974 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:15.285213 11974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:15.285588 11974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:15.288566 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "937f71eb4523425a8de0bde2c872a706"
format_stamp: "Formatted at 2026-09-24 19:12:15 on dist-test-slave-0lgr"
server_key: "94507847136a51d842e4bf2f65704cd0"
server_key_iv: "31597e6089e20dbd82d06aec14703d3c"
server_key_version: "encryptionkey@0"
I20260924 19:12:15.289515 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "937f71eb4523425a8de0bde2c872a706"
format_stamp: "Formatted at 2026-09-24 19:12:15 on dist-test-slave-0lgr"
server_key: "94507847136a51d842e4bf2f65704cd0"
server_key_iv: "31597e6089e20dbd82d06aec14703d3c"
server_key_version: "encryptionkey@0"
I20260924 19:12:15.295014 11974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:12:15.299005 11989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:15.300575 11974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:15.300866 11974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "937f71eb4523425a8de0bde2c872a706"
format_stamp: "Formatted at 2026-09-24 19:12:15 on dist-test-slave-0lgr"
server_key: "94507847136a51d842e4bf2f65704cd0"
server_key_iv: "31597e6089e20dbd82d06aec14703d3c"
server_key_version: "encryptionkey@0"
I20260924 19:12:15.301177 11974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:15.318645 11974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:15.323421 11974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:15.323771 11974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:15.324887 11974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:15.326344 11974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:15.326447 11974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:15.326593 11974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:15.326668 11974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:15.363915 11974 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:41555
I20260924 19:12:15.363970 12101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:41555 every 8 connection(s)
I20260924 19:12:15.365437 11974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:15.368180 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 11974
I20260924 19:12:15.368453 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:15.369074 11337 external_mini_cluster.cc:1428] Setting key be7a526d39407bf268ce95054f5a66fa
Sep 24 19:12:15 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277135, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
I20260924 19:12:15.390933 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:15.368378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:49863 (local address 127.11.18.126:45333)
0924 19:12:15.369690 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:15.369699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:15.370514 (+ 815us) server_negotiation.cc:407] Connection header received
0924 19:12:15.373589 (+ 3075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:15.373594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:15.373663 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:15.373763 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:15.376438 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:15.377831 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:15.379554 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:15.380338 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:15.380488 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:15.384907 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:15.384913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:15.384917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:15.384952 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:15.387677 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:15.388472 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:15.388479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:15.388482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:15.388616 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:15.389050 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:15.389064 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:15.389067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:15.389453 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:15.389614 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:15.390349 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 19:12:15.390571 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1183,"thread_start_us":1076,"threads_started":1}
I20260924 19:12:15.392017 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:15.369274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45333 (local address 127.11.18.65:49863)
0924 19:12:15.370327 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:15.370384 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:15.373178 (+ 2794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:15.374099 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:15.374115 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:15.375074 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:15.376192 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:15.376224 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:15.378043 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:15.378047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:15.379400 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:15.379412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:15.380592 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:15.380597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:15.380824 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:15.381708 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:15.381732 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:15.384567 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:15.387844 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:15.387854 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:15.387873 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:15.388342 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:15.388731 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:15.388736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:15.388740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:15.388940 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:15.389609 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:15.389617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:15.389972 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 19:12:15.390298 (+ 326us) client_negotiation.cc:240] Negotiation successful
0924 19:12:15.390862 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":410,"threads_started":1}
I20260924 19:12:15.394279 12102 heartbeater.cc:347] Connected to a master server at 127.11.18.126:45333
I20260924 19:12:15.394649 12102 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:15.395483 12102 heartbeater.cc:511] Master 127.11.18.126:45333 requested a full tablet report, sending...
I20260924 19:12:15.397913 11913 ts_manager.cc:194] Registered new tserver with Master: 937f71eb4523425a8de0bde2c872a706 (127.11.18.65:41555)
I20260924 19:12:15.399403 11913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:49863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:12:15.418095 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45333
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:15.770906 12108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:15.771248 12108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:15.771421 12108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:15.779105 12108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:15.779268 12108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:15.779458 12108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:15.789558 12108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:15.790870 12108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:15.792223 12108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:15.805967 12114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:15.806006 12116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:15.806046 12113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:15.807552 12108 server_base.cc:1034] running on GCE node
I20260924 19:12:15.808238 12108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:15.809482 12108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:15.810747 12108 hybrid_clock.cc:648] HybridClock initialized: now 1790277135810674 us; error 60 us; skew 500 ppm
I20260924 19:12:15.811265 12108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:15 dist-test-slave-0lgr krb5kdc[11868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277135, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:15.819471 12108 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:12:15.821416 12108 webserver.cc:460] Webserver started at http://127.11.18.66:42171/ using document root <none> and password file <none>
I20260924 19:12:15.822211 12108 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:15.822376 12108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:15.822729 12108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:15.826366 12108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "3fc041ee4d5547228a78208645d463ce"
format_stamp: "Formatted at 2026-09-24 19:12:15 on dist-test-slave-0lgr"
server_key: "de9aa74b27932bd4359e972d8dcffead"
server_key_iv: "2b43a4ff73f70d38f132ed6be5c41b0f"
server_key_version: "encryptionkey@0"
I20260924 19:12:15.827338 12108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "3fc041ee4d5547228a78208645d463ce"
format_stamp: "Formatted at 2026-09-24 19:12:15 on dist-test-slave-0lgr"
server_key: "de9aa74b27932bd4359e972d8dcffead"
server_key_iv: "2b43a4ff73f70d38f132ed6be5c41b0f"
server_key_version: "encryptionkey@0"
I20260924 19:12:15.833189 12108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 19:12:15.837534 12123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:15.839100 12108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:15.839313 12108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "3fc041ee4d5547228a78208645d463ce"
format_stamp: "Formatted at 2026-09-24 19:12:15 on dist-test-slave-0lgr"
server_key: "de9aa74b27932bd4359e972d8dcffead"
server_key_iv: "2b43a4ff73f70d38f132ed6be5c41b0f"
server_key_version: "encryptionkey@0"
I20260924 19:12:15.839535 12108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:15.853309 12108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:15.858441 12108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:15.858785 12108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:15.859898 12108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:15.861423 12108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:15.861531 12108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:15.861692 12108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:15.861773 12108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:15.899845 12108 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:32983
I20260924 19:12:15.899889 12235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:32983 every 8 connection(s)
I20260924 19:12:15.901466 12108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:15.908645 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 12108
I20260924 19:12:15.908859 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:15.909502 11337 external_mini_cluster.cc:1428] Setting key f4b08d610db901fe1fb4bd07a7e5d487
Sep 24 19:12:15 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277135, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:15.926957 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:15.904230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:37109 (local address 127.11.18.126:45333)
0924 19:12:15.904729 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:15.904734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:15.906101 (+ 1367us) server_negotiation.cc:407] Connection header received
0924 19:12:15.908116 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:15.908123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:15.908203 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:15.908324 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:15.910735 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:15.911837 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:15.914895 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:15.915429 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:15.915540 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:15.920048 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:15.920054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:15.920058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:15.920097 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:15.923113 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:15.924234 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:15.924238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:15.924241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:15.924379 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:15.924881 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:15.924888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:15.924897 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:15.925262 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:15.925491 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:15.926272 (+ 781us) server_negotiation.cc:299] Negotiation successful
0924 19:12:15.926524 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":231,"threads_started":1}
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:12:15.928092 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:15.904884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45333 (local address 127.11.18.66:37109)
0924 19:12:15.905914 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:15.905955 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:15.907803 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:15.908627 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:15.908649 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:15.909589 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:15.910463 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:15.910489 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:15.912032 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:15.912040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:15.914712 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:15.914734 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:15.915698 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:15.915709 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:15.916091 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:15.916973 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:15.917013 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:15.919745 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:15.923291 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:15.923304 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:15.923325 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:15.924068 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:15.924511 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:15.924518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:15.924522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:15.924759 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:15.925456 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:15.925466 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:15.925871 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 19:12:15.926172 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 19:12:15.926811 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":523,"threads_started":1}
I20260924 19:12:15.929939 12236 heartbeater.cc:347] Connected to a master server at 127.11.18.126:45333
I20260924 19:12:15.930373 12236 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:15.931118 12236 heartbeater.cc:511] Master 127.11.18.126:45333 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:12:15.932803 11913 ts_manager.cc:194] Registered new tserver with Master: 3fc041ee4d5547228a78208645d463ce (127.11.18.66:32983)
I20260924 19:12:15.933732 11913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:37109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:12:15.958479 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45333
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:16.305999 12242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:16.306303 12242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:16.306454 12242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:16.314083 12242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:16.314213 12242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:16.314443 12242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:16.324659 12242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:16.326035 12242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:16.327390 12242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:16.341368 12247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:16.341272 12250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:16.341405 12248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:16.342109 12242 server_base.cc:1034] running on GCE node
I20260924 19:12:16.342832 12242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:16.343962 12242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:16.345306 12242 hybrid_clock.cc:648] HybridClock initialized: now 1790277136345209 us; error 110 us; skew 500 ppm
I20260924 19:12:16.345700 12242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:16 dist-test-slave-0lgr krb5kdc[11868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277136, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:16.349645 12242 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:12:16.351305 12242 webserver.cc:460] Webserver started at http://127.11.18.67:32797/ using document root <none> and password file <none>
I20260924 19:12:16.352111 12242 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:16.352279 12242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:16.352648 12242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:16.355902 12242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "06fc3cbcefd44805aba02e4e306cf9dd"
format_stamp: "Formatted at 2026-09-24 19:12:16 on dist-test-slave-0lgr"
server_key: "28cbaf2de170e90d92770e06fc904c7c"
server_key_iv: "3663cd6e058a1eb9c65b5af1d23b466a"
server_key_version: "encryptionkey@0"
I20260924 19:12:16.356804 12242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "06fc3cbcefd44805aba02e4e306cf9dd"
format_stamp: "Formatted at 2026-09-24 19:12:16 on dist-test-slave-0lgr"
server_key: "28cbaf2de170e90d92770e06fc904c7c"
server_key_iv: "3663cd6e058a1eb9c65b5af1d23b466a"
server_key_version: "encryptionkey@0"
I20260924 19:12:16.362411 12242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 19:12:16.367009 12257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:16.368729 12242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:12:16.368892 12242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "06fc3cbcefd44805aba02e4e306cf9dd"
format_stamp: "Formatted at 2026-09-24 19:12:16 on dist-test-slave-0lgr"
server_key: "28cbaf2de170e90d92770e06fc904c7c"
server_key_iv: "3663cd6e058a1eb9c65b5af1d23b466a"
server_key_version: "encryptionkey@0"
I20260924 19:12:16.369109 12242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:16.393972 12242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:16.398614 12242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:16.398924 12242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:16.399966 12242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:16.401546 12242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:16.401621 12242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:16.401750 12242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:16.401800 12242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:16.402902 12102 heartbeater.cc:503] Master 127.11.18.126:45333 was elected leader, sending a full tablet report...
I20260924 19:12:16.439782 12242 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:34491
I20260924 19:12:16.439831 12369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:34491 every 8 connection(s)
I20260924 19:12:16.441140 12242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:16.448421 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 12242
I20260924 19:12:16.448683 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:16.449115 11337 external_mini_cluster.cc:1428] Setting key 02e18507cb5ac327b85d242cd6ba6656
Sep 24 19:12:16 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277136, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:16.462857 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.444068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:47013 (local address 127.11.18.126:45333)
0924 19:12:16.444537 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:16.444542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:16.445822 (+ 1280us) server_negotiation.cc:407] Connection header received
0924 19:12:16.447721 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:16.447727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:16.447821 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:16.447942 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:16.449750 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.450807 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.452597 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.453001 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.453160 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.456746 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:16.456750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:16.456754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:16.456786 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:16.459656 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:16.460467 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:16.460478 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:16.460480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:16.460598 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:16.461034 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:16.461039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:16.461042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:16.461314 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:16.461520 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:16.462255 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 19:12:16.462478 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":255,"threads_started":1}
I20260924 19:12:16.464047 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.444474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45333 (local address 127.11.18.67:47013)
0924 19:12:16.445650 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:16.445696 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:16.447429 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:16.448208 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:16.448217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:16.448859 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:16.449551 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.449571 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.450960 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.450964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.452448 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.452462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.453264 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.453271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.453552 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.454270 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:16.454294 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:16.456507 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:16.459827 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:16.459837 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:16.459857 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:16.460328 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:16.460709 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:16.460714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:16.460718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:16.460912 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:16.461492 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:16.461501 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:16.461854 (+ 353us) client_negotiation.cc:769] Sending connection context
0924 19:12:16.462137 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 19:12:16.462676 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":262,"threads_started":1}
I20260924 19:12:16.466046 12370 heartbeater.cc:347] Connected to a master server at 127.11.18.126:45333
I20260924 19:12:16.466322 12370 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:16.466971 12370 heartbeater.cc:511] Master 127.11.18.126:45333 requested a full tablet report, sending...
I20260924 19:12:16.468386 11913 ts_manager.cc:194] Registered new tserver with Master: 06fc3cbcefd44805aba02e4e306cf9dd (127.11.18.67:34491)
I20260924 19:12:16.469142 11913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:47013
I20260924 19:12:16.477645 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:12:16.498170 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.484158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.11.18.126:45333)
0924 19:12:16.484421 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:16.484432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:16.484875 (+ 443us) server_negotiation.cc:407] Connection header received
0924 19:12:16.485215 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:16.485219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:16.485286 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:16.485452 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:16.487020 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.487818 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.489221 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.489766 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.489877 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.492610 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:16.492615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:16.492619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:16.492654 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:16.495514 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:16.496212 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:16.496216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:16.496219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:16.496411 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:16.496843 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:16.496846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:16.496848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:16.497151 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:16.497463 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:16.497775 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 19:12:16.497937 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 19:12:16.505254 11913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:12:16.512264 11913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:16.538928 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.525896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.11.18.65:41555)
0924 19:12:16.526748 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:16.526761 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:16.527413 (+ 652us) server_negotiation.cc:407] Connection header received
0924 19:12:16.527816 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:16.527827 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:16.528095 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:16.528254 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:16.529398 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.531404 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.534902 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.536844 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.536991 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.537277 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:16.538190 (+ 913us) server_negotiation.cc:299] Negotiation successful
0924 19:12:16.538434 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":634,"threads_started":1}
I20260924 19:12:16.539902 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.525768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:41555 (local address 127.0.0.1:60562)
0924 19:12:16.527278 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:16.527311 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:16.527555 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:16.528632 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:16.528643 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:16.528684 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:16.529147 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.529169 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.531625 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.531632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.534692 (+ 3060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.534712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.537550 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.537560 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.537871 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.537943 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 19:12:16.539083 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0924 19:12:16.539382 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1285,"thread_start_us":1150,"threads_started":1}
I20260924 19:12:16.545692 12037 tablet_service.cc:1467] Processing CreateTablet for tablet a0fd726efa7448a496e148606aa1b5d8 (DEFAULT_TABLE table=test-table [id=19b1eea6029c467abc38181197d03e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:16.547322 12037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fd726efa7448a496e148606aa1b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:16.548130 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.527746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:34491 (local address 127.0.0.1:56478)
0924 19:12:16.540640 (+ 12894us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:16.540658 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:16.540779 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:16.541952 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:16.541955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:16.541977 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:16.542102 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.542109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.543980 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.543985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.545853 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.545865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.547197 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.547203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.547404 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.547536 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:12:16.547627 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:12:16.547783 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12807,"thread_start_us":12887,"threads_started":1}
I20260924 19:12:16.548993 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.528245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.11.18.67:34491)
0924 19:12:16.540390 (+ 12145us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:16.540403 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:16.540899 (+ 496us) server_negotiation.cc:407] Connection header received
0924 19:12:16.541029 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:16.541041 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:16.541445 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:16.541681 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:16.542261 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.543774 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.546016 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.547025 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.547179 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.547435 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:16.548462 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0924 19:12:16.548636 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11944,"thread_start_us":11791,"threads_started":1}
I20260924 19:12:16.550009 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.526346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:32983 (local address 127.0.0.1:42906)
0924 19:12:16.537721 (+ 11375us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:16.537750 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:16.537919 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:16.542467 (+ 4548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:16.542473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:16.542502 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:16.542837 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.542848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.545094 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.545100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.547174 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.547189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.549004 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.549013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.549439 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.549464 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:12:16.549566 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:12:16.549694 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11181,"thread_start_us":11708,"threads_started":1}
I20260924 19:12:16.552217 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.526650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.11.18.66:32983)
0924 19:12:16.541923 (+ 15273us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:16.541931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:16.541958 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:12:16.542051 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:16.542059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:16.542275 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:16.542471 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:16.543024 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.544848 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.547374 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.548791 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.548939 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.549116 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:16.549611 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 19:12:16.549792 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15123,"thread_start_us":14998,"threads_started":1}
I20260924 19:12:16.552783 12305 tablet_service.cc:1467] Processing CreateTablet for tablet a0fd726efa7448a496e148606aa1b5d8 (DEFAULT_TABLE table=test-table [id=19b1eea6029c467abc38181197d03e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:16.554337 12305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fd726efa7448a496e148606aa1b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:16.556474 12171 tablet_service.cc:1467] Processing CreateTablet for tablet a0fd726efa7448a496e148606aa1b5d8 (DEFAULT_TABLE table=test-table [id=19b1eea6029c467abc38181197d03e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:16.558604 12171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fd726efa7448a496e148606aa1b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:16.560325 12389 tablet_bootstrap.cc:492] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706: Bootstrap starting.
I20260924 19:12:16.564289 12389 tablet_bootstrap.cc:654] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:16.565416 12389 log.cc:824] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:16.569603 12391 tablet_bootstrap.cc:492] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd: Bootstrap starting.
I20260924 19:12:16.571636 12389 tablet_bootstrap.cc:492] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706: No bootstrap required, opened a new log
I20260924 19:12:16.572214 12389 ts_tablet_manager.cc:1397] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20260924 19:12:16.573854 12392 tablet_bootstrap.cc:492] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce: Bootstrap starting.
I20260924 19:12:16.574189 12391 tablet_bootstrap.cc:654] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:16.575630 12391 log.cc:824] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:16.577170 12392 tablet_bootstrap.cc:654] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:16.578869 12392 log.cc:824] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:16.578986 12391 tablet_bootstrap.cc:492] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd: No bootstrap required, opened a new log
I20260924 19:12:16.579327 12391 ts_tablet_manager.cc:1397] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 19:12:16.584050 12392 tablet_bootstrap.cc:492] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce: No bootstrap required, opened a new log
I20260924 19:12:16.584321 12392 ts_tablet_manager.cc:1397] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260924 19:12:16.584620 12389 raft_consensus.cc:348] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.585103 12389 raft_consensus.cc:374] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:16.585215 12389 raft_consensus.cc:729] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937f71eb4523425a8de0bde2c872a706, State: Initialized, Role: FOLLOWER
I20260924 19:12:16.585924 12389 consensus_queue.cc:260] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.587857 12389 ts_tablet_manager.cc:1428] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260924 19:12:16.590490 12391 raft_consensus.cc:348] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.590981 12391 raft_consensus.cc:374] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:16.591106 12391 raft_consensus.cc:729] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06fc3cbcefd44805aba02e4e306cf9dd, State: Initialized, Role: FOLLOWER
I20260924 19:12:16.591794 12391 consensus_queue.cc:260] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.592918 12370 heartbeater.cc:503] Master 127.11.18.126:45333 was elected leader, sending a full tablet report...
I20260924 19:12:16.593714 12392 raft_consensus.cc:348] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.594522 12392 raft_consensus.cc:374] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:16.594651 12392 raft_consensus.cc:729] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc041ee4d5547228a78208645d463ce, State: Initialized, Role: FOLLOWER
I20260924 19:12:16.595160 12391 ts_tablet_manager.cc:1428] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd: Time spent starting tablet: real 0.016s user 0.006s sys 0.008s
I20260924 19:12:16.595271 12392 consensus_queue.cc:260] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.596455 12236 heartbeater.cc:503] Master 127.11.18.126:45333 was elected leader, sending a full tablet report...
I20260924 19:12:16.597041 12392 ts_tablet_manager.cc:1428] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260924 19:12:16.618299 12103 tablet.cc:2369] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:12:16.653774 12237 tablet.cc:2369] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:12:16.693120 12371 tablet.cc:2369] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:12:16.939584 12397 raft_consensus.cc:482] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:16.939987 12397 raft_consensus.cc:504] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.941960 12397 leader_election.cc:288] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 937f71eb4523425a8de0bde2c872a706 (127.11.18.65:41555), 06fc3cbcefd44805aba02e4e306cf9dd (127.11.18.67:34491)
I20260924 19:12:16.950220 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.942400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:55163 (local address 127.11.18.65:41555)
0924 19:12:16.942671 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:16.942679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:16.943834 (+ 1155us) server_negotiation.cc:407] Connection header received
0924 19:12:16.944101 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:16.944107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:16.944196 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:16.944312 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:16.945004 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.945962 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.948308 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.949111 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.949297 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.949422 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:16.949843 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 19:12:16.949971 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:12:16.950222 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.943488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:45267 (local address 127.11.18.67:34491)
0924 19:12:16.943721 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:16.943727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:16.944250 (+ 523us) server_negotiation.cc:407] Connection header received
0924 19:12:16.944475 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:16.944481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:16.944552 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:16.944665 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:16.945434 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.946379 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.948262 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.949124 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.949352 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.949432 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:16.949793 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 19:12:16.949939 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:12:16.950362 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.943080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:34491 (local address 127.11.18.66:45267)
0924 19:12:16.944031 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:16.944052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:16.944312 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:16.944680 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:16.944684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:16.944715 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:16.945213 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.945224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.946534 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.946542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.948067 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.948088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.949417 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.949424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.949607 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.949650 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:12:16.949732 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:12:16.949899 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":8,"thread_start_us":547,"threads_started":1}
I20260924 19:12:16.950656 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:16.942207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:41555 (local address 127.11.18.66:55163)
0924 19:12:16.943722 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:16.943747 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:16.943922 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:16.944358 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:16.944365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:16.944396 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:16.944816 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.944826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.946206 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.946212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.948074 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:16.948094 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:16.949368 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:16.949376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:16.949650 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:16.949675 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:12:16.949799 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 19:12:16.949939 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"mutex_wait_us":103,"thread_start_us":392,"threads_started":1}
I20260924 19:12:16.951424 12325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fd726efa7448a496e148606aa1b5d8" candidate_uuid: "3fc041ee4d5547228a78208645d463ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" is_pre_election: true
I20260924 19:12:16.951757 12057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fd726efa7448a496e148606aa1b5d8" candidate_uuid: "3fc041ee4d5547228a78208645d463ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "937f71eb4523425a8de0bde2c872a706" is_pre_election: true
I20260924 19:12:16.951891 12325 raft_consensus.cc:2454] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc041ee4d5547228a78208645d463ce in term 0.
I20260924 19:12:16.952292 12057 raft_consensus.cc:2454] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc041ee4d5547228a78208645d463ce in term 0.
I20260924 19:12:16.952620 12126 leader_election.cc:302] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06fc3cbcefd44805aba02e4e306cf9dd, 3fc041ee4d5547228a78208645d463ce; no voters:
I20260924 19:12:16.953128 12397 raft_consensus.cc:2781] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:16.953365 12397 raft_consensus.cc:482] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:16.953495 12397 raft_consensus.cc:3037] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:16.956180 12397 raft_consensus.cc:504] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.956825 12397 leader_election.cc:288] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [CANDIDATE]: Term 1 election: Requested vote from peers 937f71eb4523425a8de0bde2c872a706 (127.11.18.65:41555), 06fc3cbcefd44805aba02e4e306cf9dd (127.11.18.67:34491)
I20260924 19:12:16.957477 12057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fd726efa7448a496e148606aa1b5d8" candidate_uuid: "3fc041ee4d5547228a78208645d463ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "937f71eb4523425a8de0bde2c872a706"
I20260924 19:12:16.957602 12325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fd726efa7448a496e148606aa1b5d8" candidate_uuid: "3fc041ee4d5547228a78208645d463ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06fc3cbcefd44805aba02e4e306cf9dd"
I20260924 19:12:16.957738 12057 raft_consensus.cc:3037] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:16.957870 12325 raft_consensus.cc:3037] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:16.960048 12057 raft_consensus.cc:2454] T a0fd726efa7448a496e148606aa1b5d8 P 937f71eb4523425a8de0bde2c872a706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc041ee4d5547228a78208645d463ce in term 1.
I20260924 19:12:16.960371 12325 raft_consensus.cc:2454] T a0fd726efa7448a496e148606aa1b5d8 P 06fc3cbcefd44805aba02e4e306cf9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc041ee4d5547228a78208645d463ce in term 1.
I20260924 19:12:16.960556 12126 leader_election.cc:302] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc041ee4d5547228a78208645d463ce, 937f71eb4523425a8de0bde2c872a706; no voters:
I20260924 19:12:16.960938 12397 raft_consensus.cc:2781] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:16.961519 12397 raft_consensus.cc:686] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [term 1 LEADER]: Becoming Leader. State: Replica: 3fc041ee4d5547228a78208645d463ce, State: Running, Role: LEADER
I20260924 19:12:16.962018 12397 consensus_queue.cc:237] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } }
I20260924 19:12:16.966630 11913 catalog_manager.cc:5495] T a0fd726efa7448a496e148606aa1b5d8 P 3fc041ee4d5547228a78208645d463ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc041ee4d5547228a78208645d463ce (127.11.18.66). New cstate: current_term: 1 leader_uuid: "3fc041ee4d5547228a78208645d463ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "937f71eb4523425a8de0bde2c872a706" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc041ee4d5547228a78208645d463ce" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06fc3cbcefd44805aba02e4e306cf9dd" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 34491 } health_report { overall_health: UNKNOWN } } }
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:17.019842 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:17.002044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.11.18.65:41555)
0924 19:12:17.002279 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:17.002288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:17.002736 (+ 448us) server_negotiation.cc:407] Connection header received
0924 19:12:17.002912 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:17.002917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:17.002988 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:17.003107 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:17.005306 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.006176 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.007683 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.008163 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.008292 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:17.013314 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:17.013405 (+ 91us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:17.013423 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:17.013467 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:17.016854 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.017713 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.017722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.017724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.017894 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.018297 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.018301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.018304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.018670 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:17.018920 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:17.019228 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 19:12:17.019399 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[11868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277137, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277137, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:17.071820 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:17.055017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.11.18.65:41555)
0924 19:12:17.055328 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:17.055336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:17.056275 (+ 939us) server_negotiation.cc:407] Connection header received
0924 19:12:17.056535 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:17.056540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:17.056611 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:17.056733 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:17.058326 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.059180 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.060715 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.061146 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.061288 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:17.065786 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:17.065793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:17.065797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:17.065836 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:17.068771 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.069680 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.069691 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.069694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.069830 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.070297 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.070310 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.070312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.070583 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:17.070898 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:17.071196 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 19:12:17.071446 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[11868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:17.124413 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:17.107131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.11.18.65:41555)
0924 19:12:17.107389 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:17.107397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:17.107793 (+ 396us) server_negotiation.cc:407] Connection header received
0924 19:12:17.107913 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:17.107917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:17.107986 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:17.108066 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:17.109940 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.111040 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.112491 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.112962 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.113075 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:17.118100 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:17.118106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:17.118110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:17.118189 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:17.120894 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.121992 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.121998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.122001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.122172 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.122654 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.122658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.122661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.122985 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:17.123187 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:17.123763 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 19:12:17.124023 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 19:12:17.126374 12017 tablet_service.cc:2908] Rejecting scan request for tablet a0fd726efa7448a496e148606aa1b5d8: Uninitialized: clean time has not yet been initialized
I20260924 19:12:17.130344 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 11974
I20260924 19:12:17.146921 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:17.131315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.11.18.65:41555)
0924 19:12:17.131552 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:17.131558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:17.131782 (+ 224us) server_negotiation.cc:407] Connection header received
0924 19:12:17.131913 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:17.131917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:17.131983 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:17.132069 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:17.133430 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.134561 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.135873 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.136396 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.136524 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:17.139517 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:17.139534 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:17.139539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:17.139580 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:17.143068 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.144004 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.144009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.144011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.144348 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.144971 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.144976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.144978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.145528 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:17.145809 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:17.146136 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 19:12:17.146418 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:12:17.147821 12097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:17.245028 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 11974
I20260924 19:12:17.264403 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12108
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:12:17.281189 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:17.265231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.11.18.66:32983)
0924 19:12:17.265696 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:17.265700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:17.265716 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:12:17.265775 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:17.265779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:17.265835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:17.265924 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:17.267175 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.267957 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.269478 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.269968 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.270068 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:17.274691 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:17.274706 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:17.274722 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:17.274757 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:17.278106 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.278968 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.278976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.278978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.279148 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.279597 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.279600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.279602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.279989 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:17.280226 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:17.280569 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 19:12:17.280777 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":273,"threads_started":1}
I20260924 19:12:17.282172 12231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:17.380314 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 12108
I20260924 19:12:17.398268 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12242
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[11868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:12:17.414743 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:17.399095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.11.18.67:34491)
0924 19:12:17.399305 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:17.399310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:17.399329 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:12:17.399442 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:17.399446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:17.399513 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:17.399651 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:17.400888 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.401768 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.403100 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:17.403526 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:17.403629 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:17.408008 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:17.408029 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:17.408054 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:17.408098 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:17.411317 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.412210 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.412218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.412221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.412372 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:17.412867 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:17.412872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:17.412874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:17.413318 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:17.413700 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:17.414086 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 19:12:17.414264 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:12:17.415493 12365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:17.505446 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 12242
I20260924 19:12:17.523767 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 11881
I20260924 19:12:17.525044 11943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:17.632944 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 11881
2026-09-24T19:12:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4653 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[12420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[12420](info): set up 2 sockets
Sep 24 19:12:17 dist-test-slave-0lgr krb5kdc[12420](info): commencing operation
krb5kdc: starting...
W20260924 19:12:18.737563 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:18 dist-test-slave-0lgr krb5kdc[12420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:18.938758 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:37703
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:19.284406 12433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:19.284759 12433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:19.284929 12433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:19.292290 12433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:19.292446 12433 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:12:19.292543 12433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:19.292659 12433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:19.292752 12433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:19.292840 12433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:19.303453 12433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:37703
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:37703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:19.304951 12433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:19.306406 12433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:19.320009 12441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:19.319937 12439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:19.319976 12438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:19.321480 12433 server_base.cc:1034] running on GCE node
I20260924 19:12:19.322311 12433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:19.323859 12433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:19.325150 12433 hybrid_clock.cc:648] HybridClock initialized: now 1790277139325086 us; error 42 us; skew 500 ppm
I20260924 19:12:19.325573 12433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:19 dist-test-slave-0lgr krb5kdc[12420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277139, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:19.329000 12433 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:19.330814 12433 webserver.cc:460] Webserver started at http://127.11.18.126:34933/ using document root <none> and password file <none>
I20260924 19:12:19.331588 12433 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:19.331729 12433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:19.332077 12433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:19.335325 12433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "aa5a50820cd843aaa91acb1f0decffc7"
format_stamp: "Formatted at 2026-09-24 19:12:19 on dist-test-slave-0lgr"
server_key: "c2c185395aae9593bb10e468c7a625ea"
server_key_iv: "ee1de6ad7345951fbe095fa2a7eea452"
server_key_version: "encryptionkey@0"
I20260924 19:12:19.336211 12433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "aa5a50820cd843aaa91acb1f0decffc7"
format_stamp: "Formatted at 2026-09-24 19:12:19 on dist-test-slave-0lgr"
server_key: "c2c185395aae9593bb10e468c7a625ea"
server_key_iv: "ee1de6ad7345951fbe095fa2a7eea452"
server_key_version: "encryptionkey@0"
I20260924 19:12:19.342108 12433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:12:19.346294 12448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:19.347999 12433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 19:12:19.348237 12433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "aa5a50820cd843aaa91acb1f0decffc7"
format_stamp: "Formatted at 2026-09-24 19:12:19 on dist-test-slave-0lgr"
server_key: "c2c185395aae9593bb10e468c7a625ea"
server_key_iv: "ee1de6ad7345951fbe095fa2a7eea452"
server_key_version: "encryptionkey@0"
I20260924 19:12:19.348485 12433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:19.366115 12433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:19.371090 12433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:19.371502 12433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:19.395942 12433 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:37703
I20260924 19:12:19.396095 12509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:37703 every 8 connection(s)
I20260924 19:12:19.397498 12433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:19.399013 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 12433
I20260924 19:12:19.399248 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:19.400030 11337 external_mini_cluster.cc:1428] Setting key e8ebaf137084bfb9913ace42ed8c0fc0
I20260924 19:12:19.402565 12510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:19.416285 12510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7: Bootstrap starting.
Sep 24 19:12:19 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:19.420663 12510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:19.422098 12510 log.cc:824] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:19.425825 12510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7: No bootstrap required, opened a new log
I20260924 19:12:19.426728 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:19.402393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.11.18.126:37703)
0924 19:12:19.403230 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:19.403245 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:19.403294 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:12:19.405571 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:19.405600 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:19.406143 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:19.406604 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:19.408104 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:19.410198 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:19.411669 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:19.412788 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:19.412990 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:19.417738 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:19.417747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:19.417763 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:19.417806 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:19.421809 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:19.422815 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:19.422833 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:19.422844 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:19.423090 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:19.423642 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:19.423646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:19.423648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:19.424250 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:19.424471 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:19.425013 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 19:12:19.425522 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":400,"threads_started":1}
I20260924 19:12:19.435487 12510 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5a50820cd843aaa91acb1f0decffc7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37703 } }
I20260924 19:12:19.435912 12510 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:19.436041 12510 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5a50820cd843aaa91acb1f0decffc7, State: Initialized, Role: FOLLOWER
I20260924 19:12:19.436663 12510 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5a50820cd843aaa91acb1f0decffc7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37703 } }
I20260924 19:12:19.436903 12510 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:19.437074 12510 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:19.437268 12510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:19.440155 12510 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5a50820cd843aaa91acb1f0decffc7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37703 } }
I20260924 19:12:19.440760 12510 leader_election.cc:302] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa5a50820cd843aaa91acb1f0decffc7; no voters:
I20260924 19:12:19.441519 12510 leader_election.cc:288] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:19.441617 12515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:19.442494 12515 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [term 1 LEADER]: Becoming Leader. State: Replica: aa5a50820cd843aaa91acb1f0decffc7, State: Running, Role: LEADER
I20260924 19:12:19.443063 12515 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5a50820cd843aaa91acb1f0decffc7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37703 } }
I20260924 19:12:19.443905 12510 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:19.445914 12517 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa5a50820cd843aaa91acb1f0decffc7. Latest consensus state: current_term: 1 leader_uuid: "aa5a50820cd843aaa91acb1f0decffc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5a50820cd843aaa91acb1f0decffc7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37703 } } }
I20260924 19:12:19.446317 12517 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:19.446785 12516 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa5a50820cd843aaa91acb1f0decffc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa5a50820cd843aaa91acb1f0decffc7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37703 } } }
I20260924 19:12:19.447026 12516 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:19.452929 12522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:19.458101 12522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:19.467037 12522 catalog_manager.cc:1347] Generated new cluster ID: 70d32f33c6fb47d4b71c4aa6eaf1fd2a
I20260924 19:12:19.467306 12522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:19.476929 12522 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:19.478456 12522 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:19.488552 12522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7: Generated new TSK 0
I20260924 19:12:19.489557 12522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:12:19.550927 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37703
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:19.893950 12536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:19.894272 12536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:19.894384 12536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:19.902829 12536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:19.902958 12536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:19.903045 12536 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:12:19.903203 12536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:19.913599 12536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:37703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:19.914989 12536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:19.916288 12536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:19.931360 12541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:19.931664 12544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:19.931706 12542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:19.932926 12536 server_base.cc:1034] running on GCE node
I20260924 19:12:19.933736 12536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:19.934897 12536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:19.936157 12536 hybrid_clock.cc:648] HybridClock initialized: now 1790277139936061 us; error 69 us; skew 500 ppm
I20260924 19:12:19.936496 12536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:19 dist-test-slave-0lgr krb5kdc[12420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277139, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:19.940208 12536 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:12:19.941874 12536 webserver.cc:460] Webserver started at http://127.11.18.65:34843/ using document root <none> and password file <none>
I20260924 19:12:19.942623 12536 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:19.942718 12536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:19.943001 12536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:19.946413 12536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "eba458d46ffe4bca9a95a75dd13ee0a8"
format_stamp: "Formatted at 2026-09-24 19:12:19 on dist-test-slave-0lgr"
server_key: "3d9a462c425bee4c9fce81898399c92d"
server_key_iv: "60c856e6760cbfff28b6ea0debbda583"
server_key_version: "encryptionkey@0"
I20260924 19:12:19.947258 12536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "eba458d46ffe4bca9a95a75dd13ee0a8"
format_stamp: "Formatted at 2026-09-24 19:12:19 on dist-test-slave-0lgr"
server_key: "3d9a462c425bee4c9fce81898399c92d"
server_key_iv: "60c856e6760cbfff28b6ea0debbda583"
server_key_version: "encryptionkey@0"
I20260924 19:12:19.953572 12536 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 19:12:19.957504 12551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:19.959149 12536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:19.959348 12536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "eba458d46ffe4bca9a95a75dd13ee0a8"
format_stamp: "Formatted at 2026-09-24 19:12:19 on dist-test-slave-0lgr"
server_key: "3d9a462c425bee4c9fce81898399c92d"
server_key_iv: "60c856e6760cbfff28b6ea0debbda583"
server_key_version: "encryptionkey@0"
I20260924 19:12:19.959594 12536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:20.006979 12536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:20.012619 12536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:20.012995 12536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:20.015980 12536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:20.016081 12536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:20.016259 12536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:20.016309 12536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:12:20 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277139, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:20.045713 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:20.016966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:50583 (local address 127.11.18.126:37703)
0924 19:12:20.017508 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:20.017515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:20.018922 (+ 1407us) server_negotiation.cc:407] Connection header received
0924 19:12:20.022481 (+ 3559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:20.022488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:20.022603 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:20.022782 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:20.025764 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.027217 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.031073 (+ 3856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.031573 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.031715 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:20.038016 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:20.038028 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:20.038036 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:20.038093 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:20.042123 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:20.043006 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:20.043012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:20.043016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:20.043157 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:20.043649 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:20.043654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:20.043658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:20.043986 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:20.044157 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:20.045022 (+ 865us) server_negotiation.cc:299] Negotiation successful
0924 19:12:20.045260 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":261,"threads_started":1}
I20260924 19:12:20.047412 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:20.017720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37703 (local address 127.11.18.65:50583)
0924 19:12:20.018691 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:20.018759 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:20.022042 (+ 3283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:20.023095 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:20.023110 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:20.024181 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:20.025457 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:20.025503 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.027515 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.027523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:20.030871 (+ 3348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:20.030895 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.031952 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.031963 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:20.032325 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:20.033248 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:20.033291 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:20.037651 (+ 4360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:20.042338 (+ 4687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:20.042345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:20.042360 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:20.042849 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:20.043285 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:20.043291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:20.043295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:20.043514 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:20.044140 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:20.044149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:20.044604 (+ 455us) client_negotiation.cc:769] Sending connection context
0924 19:12:20.044913 (+ 309us) client_negotiation.cc:240] Negotiation successful
0924 19:12:20.045557 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":370,"threads_started":1}
I20260924 19:12:20.070169 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:20.054582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:49679 (local address 127.11.18.126:37703)
0924 19:12:20.054767 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:20.054771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:20.055520 (+ 749us) server_negotiation.cc:407] Connection header received
0924 19:12:20.055790 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:20.055795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:20.055857 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:20.055964 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:20.056950 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.057753 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.060636 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.061421 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.061560 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:20.063814 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:20.063821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:20.063827 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:20.063866 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:20.066660 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:20.067554 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:20.067559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:20.067563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:20.067747 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:20.068318 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:20.068326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:20.068331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:20.068753 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:20.068946 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:20.069559 (+ 613us) server_negotiation.cc:299] Negotiation successful
0924 19:12:20.069797 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:12:20.070312 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:20.054677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37703 (local address 127.11.18.65:49679)
0924 19:12:20.055394 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:20.055410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:20.055641 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:20.055987 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:20.055991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:20.056510 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:20.056799 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:20.056808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.057916 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.057926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:20.060452 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:20.060472 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.061833 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.061838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:20.061957 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:20.062644 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:20.062662 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:20.063628 (+ 966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:20.066817 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:20.066822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:20.066826 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:20.067436 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:20.067907 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:20.067915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:20.067920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:20.068143 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:20.068911 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:20.068919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:20.069252 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 19:12:20.069464 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 19:12:20.069815 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":425,"threads_started":1}
I20260924 19:12:20.079448 12536 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:41843
I20260924 19:12:20.079465 12671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:41843 every 8 connection(s)
I20260924 19:12:20.080639 12536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:20.081892 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 12536
I20260924 19:12:20.082175 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:20.083098 11337 external_mini_cluster.cc:1428] Setting key 17b06c066871c466b5e4aba3a9b3e307
I20260924 19:12:20.083168 12672 heartbeater.cc:347] Connected to a master server at 127.11.18.126:37703
I20260924 19:12:20.083411 12672 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:20.084172 12672 heartbeater.cc:511] Master 127.11.18.126:37703 requested a full tablet report, sending...
I20260924 19:12:20.086375 12465 ts_manager.cc:194] Registered new tserver with Master: eba458d46ffe4bca9a95a75dd13ee0a8 (127.11.18.65:41843)
I20260924 19:12:20.088624 12465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:50583
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:12:20.130479 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37703
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:20.477128 12676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:20.477598 12676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:20.477746 12676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:20.485556 12676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:20.485673 12676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:20.485756 12676 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:12:20.485834 12676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:20.495822 12676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:37703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:20.497082 12676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:20.498512 12676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:20.512687 12684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:20.512743 12681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:20.512765 12682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:20.514312 12676 server_base.cc:1034] running on GCE node
I20260924 19:12:20.515094 12676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:20.516162 12676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:20.517491 12676 hybrid_clock.cc:648] HybridClock initialized: now 1790277140517421 us; error 32 us; skew 500 ppm
I20260924 19:12:20.517874 12676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:20 dist-test-slave-0lgr krb5kdc[12420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277140, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:20.521715 12676 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:12:20.523423 12676 webserver.cc:460] Webserver started at http://127.11.18.66:36403/ using document root <none> and password file <none>
I20260924 19:12:20.524226 12676 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:20.524372 12676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:20.524730 12676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:20.527946 12676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "6d377fe07fc34749befba9ef02d3292f"
format_stamp: "Formatted at 2026-09-24 19:12:20 on dist-test-slave-0lgr"
server_key: "7cc81537fbe2bdcc7d0a8ab3f9d4f610"
server_key_iv: "6b1d18c78213c9d6dd7bdc7bf82b1972"
server_key_version: "encryptionkey@0"
I20260924 19:12:20.528880 12676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "6d377fe07fc34749befba9ef02d3292f"
format_stamp: "Formatted at 2026-09-24 19:12:20 on dist-test-slave-0lgr"
server_key: "7cc81537fbe2bdcc7d0a8ab3f9d4f610"
server_key_iv: "6b1d18c78213c9d6dd7bdc7bf82b1972"
server_key_version: "encryptionkey@0"
I20260924 19:12:20.534812 12676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260924 19:12:20.538753 12691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:20.540323 12676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:20.540549 12676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "6d377fe07fc34749befba9ef02d3292f"
format_stamp: "Formatted at 2026-09-24 19:12:20 on dist-test-slave-0lgr"
server_key: "7cc81537fbe2bdcc7d0a8ab3f9d4f610"
server_key_iv: "6b1d18c78213c9d6dd7bdc7bf82b1972"
server_key_version: "encryptionkey@0"
I20260924 19:12:20.540818 12676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:20.563621 12676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:20.568485 12676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:20.568900 12676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:20.572508 12676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:20.572636 12676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:20.572856 12676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:20.572983 12676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:12:20 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277140, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:20.602180 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:20.572643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:48075 (local address 127.11.18.126:37703)
0924 19:12:20.573129 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:20.573136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:20.574960 (+ 1824us) server_negotiation.cc:407] Connection header received
0924 19:12:20.577902 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:20.577908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:20.578047 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:20.578185 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:20.581618 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.583092 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.586243 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.587091 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.587359 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:20.593044 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:20.593050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:20.593053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:20.593094 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:20.597679 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:20.598933 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:20.598940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:20.598944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:20.599151 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:20.599871 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:20.599878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:20.599882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:20.600204 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:20.600340 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:20.601605 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0924 19:12:20.601807 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260924 19:12:20.604612 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:20.573523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37703 (local address 127.11.18.66:48075)
0924 19:12:20.574715 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:20.574780 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:20.577458 (+ 2678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:20.578643 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:20.578661 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:20.579934 (+ 1273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:20.581266 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:20.581311 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.583394 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.583404 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:20.585960 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:20.585989 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.587493 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.587501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:20.587752 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:20.588857 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:20.588898 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:20.592730 (+ 3832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:20.597897 (+ 5167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:20.597915 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:20.597942 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:20.598747 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:20.599293 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:20.599299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:20.599307 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:20.599708 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:20.600364 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:20.600377 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:20.601066 (+ 689us) client_negotiation.cc:769] Sending connection context
0924 19:12:20.601505 (+ 439us) client_negotiation.cc:240] Negotiation successful
0924 19:12:20.602318 (+ 813us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":552,"threads_started":1}
I20260924 19:12:20.629026 12676 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:45469
I20260924 19:12:20.629134 12811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:45469 every 8 connection(s)
I20260924 19:12:20.630617 12676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:20.632880 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:20.612007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37703 (local address 127.11.18.66:45845)
0924 19:12:20.612884 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:20.612911 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:20.613073 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:20.613726 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:20.613735 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:20.614734 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:20.615383 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:20.615404 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.617387 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.617398 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:20.619649 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:20.619684 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.621409 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.621420 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:20.621724 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:20.622666 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:20.622705 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:20.624487 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:20.629500 (+ 5013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:20.629506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:20.629509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:20.629934 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:20.630419 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:20.630427 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:20.630432 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:20.630640 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:20.631606 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:20.631616 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:20.631892 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 19:12:20.632087 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 19:12:20.632482 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":478,"threads_started":1}
I20260924 19:12:20.633375 12812 heartbeater.cc:347] Connected to a master server at 127.11.18.126:37703
I20260924 19:12:20.633670 12812 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:20.633821 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:20.611817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:45845 (local address 127.11.18.126:37703)
0924 19:12:20.611988 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:20.611995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:20.613050 (+ 1055us) server_negotiation.cc:407] Connection header received
0924 19:12:20.613193 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:20.613202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:20.613312 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:20.613473 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:20.615668 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.617122 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.619933 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:20.620932 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:20.621144 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:20.624850 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:20.624862 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:20.624870 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:20.624920 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:20.629281 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:20.630078 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:20.630087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:20.630094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:20.630259 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:20.630852 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:20.630858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:20.630862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:20.631400 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:20.633089 (+ 1689us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:20.633256 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 19:12:20.633547 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:12:20.634712 12812 heartbeater.cc:511] Master 127.11.18.126:37703 requested a full tablet report, sending...
I20260924 19:12:20.636375 12465 ts_manager.cc:194] Registered new tserver with Master: 6d377fe07fc34749befba9ef02d3292f (127.11.18.66:45469)
I20260924 19:12:20.637668 12465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:48075
I20260924 19:12:20.640885 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 12676
I20260924 19:12:20.641260 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:20.641767 11337 external_mini_cluster.cc:1428] Setting key 56e23f1dd1c897e65720a099d3fedc3a
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:12:20.686398 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37703
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:21.022652 12816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:21.023074 12816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:21.023247 12816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:21.030701 12816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:21.030880 12816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:21.031037 12816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:12:21.031188 12816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:21.041604 12816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:37703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:21.043004 12816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:21.044613 12816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:21.059790 12821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:21.059760 12822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:21.059860 12824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:21.060591 12816 server_base.cc:1034] running on GCE node
I20260924 19:12:21.061295 12816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:21.062438 12816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:21.063723 12816 hybrid_clock.cc:648] HybridClock initialized: now 1790277141063628 us; error 62 us; skew 500 ppm
I20260924 19:12:21.064051 12816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:21 dist-test-slave-0lgr krb5kdc[12420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277141, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:21.073045 12816 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:12:21.074808 12816 webserver.cc:460] Webserver started at http://127.11.18.67:41455/ using document root <none> and password file <none>
I20260924 19:12:21.075569 12816 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:21.075666 12816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:21.075956 12816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:21.079427 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "c8f6cf52f8c44713a7057ceb5fc29d57"
format_stamp: "Formatted at 2026-09-24 19:12:21 on dist-test-slave-0lgr"
server_key: "4606b0ba8a0acf302efc3cd93fb0b7f1"
server_key_iv: "3accbbc93a96866d5eb3f2d5b8cb1103"
server_key_version: "encryptionkey@0"
I20260924 19:12:21.080323 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "c8f6cf52f8c44713a7057ceb5fc29d57"
format_stamp: "Formatted at 2026-09-24 19:12:21 on dist-test-slave-0lgr"
server_key: "4606b0ba8a0acf302efc3cd93fb0b7f1"
server_key_iv: "3accbbc93a96866d5eb3f2d5b8cb1103"
server_key_version: "encryptionkey@0"
I20260924 19:12:21.086201 12816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:12:21.090237 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:21.091759 12816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:21.091961 12816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "c8f6cf52f8c44713a7057ceb5fc29d57"
format_stamp: "Formatted at 2026-09-24 19:12:21 on dist-test-slave-0lgr"
server_key: "4606b0ba8a0acf302efc3cd93fb0b7f1"
server_key_iv: "3accbbc93a96866d5eb3f2d5b8cb1103"
server_key_version: "encryptionkey@0"
I20260924 19:12:21.092233 12816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:21.092379 12672 heartbeater.cc:503] Master 127.11.18.126:37703 was elected leader, sending a full tablet report...
I20260924 19:12:21.111406 12816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:21.116297 12816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:21.116611 12816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:21.119567 12816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:21.119689 12816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:21.119870 12816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:21.119978 12816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:12:21 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277141, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:21.146986 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.120226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:47065 (local address 127.11.18.126:37703)
0924 19:12:21.120498 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.120505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.122105 (+ 1600us) server_negotiation.cc:407] Connection header received
0924 19:12:21.124183 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.124192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.124318 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.124449 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:21.126929 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.127986 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.130787 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.131387 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.131527 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.136902 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:21.136912 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:21.136917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:21.137092 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:21.142002 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:21.143248 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:21.143253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:21.143256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:21.143430 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:21.144245 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:21.144249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:21.144251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:21.144644 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:21.144823 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.146246 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.146527 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 19:12:21.149181 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.120950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37703 (local address 127.11.18.67:47065)
0924 19:12:21.121901 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:21.121960 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:21.123884 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:21.124841 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:21.124849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:21.125647 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:21.126730 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.126763 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.128187 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.128190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.130559 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.130577 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.131646 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.131654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.131953 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.132766 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:21.132788 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:21.136520 (+ 3732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:21.142261 (+ 5741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:21.142276 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:21.142309 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:21.143044 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:21.143650 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:21.143662 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:21.143669 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:21.144057 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:21.144791 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:21.144805 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:21.145568 (+ 763us) client_negotiation.cc:769] Sending connection context
0924 19:12:21.146012 (+ 444us) client_negotiation.cc:240] Negotiation successful
0924 19:12:21.146882 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":405,"threads_started":1}
I20260924 19:12:21.173032 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.156463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:60985 (local address 127.11.18.126:37703)
0924 19:12:21.156670 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.156677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.157689 (+ 1012us) server_negotiation.cc:407] Connection header received
0924 19:12:21.157911 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.157916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.157995 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.158104 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:21.159136 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.159957 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.161459 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.162143 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.162298 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.165617 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:21.165623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:21.165627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:21.165661 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:21.169238 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:21.170515 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:21.170523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:21.170530 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:21.170730 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:21.171336 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:21.171342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:21.171345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:21.171652 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:21.171825 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.172345 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.172602 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:12:21.173249 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.156832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37703 (local address 127.11.18.67:60985)
0924 19:12:21.157551 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:21.157573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:21.157756 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:21.158130 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:21.158136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:21.158655 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:21.158974 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.158984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.160106 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.160112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.161258 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.161270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.162364 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.162373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.162640 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.163599 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:21.163635 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:21.165361 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:21.169478 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:21.169484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:21.169489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:21.170305 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:21.170865 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:21.170871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:21.170876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:21.171125 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:21.171797 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:21.171803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:21.172033 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 19:12:21.172235 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 19:12:21.172644 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":375,"threads_started":1}
I20260924 19:12:21.178514 12816 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:40017
I20260924 19:12:21.178553 12950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:40017 every 8 connection(s)
I20260924 19:12:21.179890 12816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:21.182252 12951 heartbeater.cc:347] Connected to a master server at 127.11.18.126:37703
I20260924 19:12:21.182502 12951 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:21.183224 12951 heartbeater.cc:511] Master 127.11.18.126:37703 requested a full tablet report, sending...
I20260924 19:12:21.184831 12465 ts_manager.cc:194] Registered new tserver with Master: c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017)
I20260924 19:12:21.185818 12465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:47065
I20260924 19:12:21.186172 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 12816
I20260924 19:12:21.186388 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:21.186887 11337 external_mini_cluster.cc:1428] Setting key 6c2c9a90a020e51a04d616f3159a9ddb
I20260924 19:12:21.190115 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:12:21 dist-test-slave-0lgr krb5kdc[12420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:12:21 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:21.239928 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.224939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.11.18.126:37703)
0924 19:12:21.225271 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.225278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.225549 (+ 271us) server_negotiation.cc:407] Connection header received
0924 19:12:21.225792 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.225797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.225860 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.225968 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:21.227370 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.228097 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.229502 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.229902 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.230047 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.234153 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:21.234158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:21.234162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:21.234196 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:21.236917 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:21.237785 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:21.237789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:21.237792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:21.237927 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:21.238391 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:21.238395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:21.238398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:21.238722 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:21.238958 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.239324 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.239575 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260924 19:12:21.244985 12465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:12:21.253252 12465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:21.294592 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.268349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:41843 (local address 127.0.0.1:55494)
0924 19:12:21.286224 (+ 17875us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:21.286244 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:21.286381 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:21.287327 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:21.287332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:21.287363 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:21.287795 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.287805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.289461 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.289467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.291327 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.291345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.293434 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.293442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.293679 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.293750 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 19:12:21.293921 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 19:12:21.294134 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17709,"mutex_wait_us":6,"thread_start_us":17498,"threads_started":1}
I20260924 19:12:21.294826 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.266951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:40017 (local address 127.0.0.1:53566)
0924 19:12:21.282187 (+ 15236us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:21.282260 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:21.282497 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:21.285749 (+ 3252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:21.285760 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:21.285807 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:21.286484 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.286497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.288578 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.288586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.291820 (+ 3234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.291836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.293793 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.293804 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.294061 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.294147 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 19:12:21.294225 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:12:21.294375 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14846,"mutex_wait_us":321,"thread_start_us":15802,"threads_started":1}
I20260924 19:12:21.295171 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.268005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.11.18.66:45469)
0924 19:12:21.283984 (+ 15979us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.284000 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.285072 (+ 1072us) server_negotiation.cc:407] Connection header received
0924 19:12:21.285234 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.285246 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.285562 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.286306 (+ 744us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:21.286880 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.288550 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.291995 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.293398 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.293531 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.293717 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.294560 (+ 843us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.294759 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15783,"thread_start_us":15659,"threads_started":1}
I20260924 19:12:21.295608 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.267487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.11.18.67:40017)
0924 19:12:21.284621 (+ 17134us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.284633 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.284675 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:12:21.284788 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.284799 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.285100 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.285344 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:21.286724 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.288424 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.292039 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.293157 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.293292 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.293627 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.295081 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.295292 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16935,"thread_start_us":16741,"threads_started":1}
I20260924 19:12:21.295657 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.266992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:45469 (local address 127.0.0.1:34530)
0924 19:12:21.284904 (+ 17912us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:21.284924 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:21.285075 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:21.285855 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:21.285863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:21.285897 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:21.286658 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.286676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.288667 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.288673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.291822 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.291836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.293872 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.293885 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.294156 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.294186 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:12:21.294280 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 19:12:21.294407 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17771,"mutex_wait_us":280,"thread_start_us":17626,"threads_started":1}
I20260924 19:12:21.295658 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.269123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.11.18.65:41843)
0924 19:12:21.285662 (+ 16539us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.285674 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.286544 (+ 870us) server_negotiation.cc:407] Connection header received
0924 19:12:21.286654 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.286664 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.286900 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.287112 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:21.287987 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.289274 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.291515 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.292891 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.293035 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.293261 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.295090 (+ 1829us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.295348 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16375,"thread_start_us":16228,"threads_started":1}
I20260924 19:12:21.299226 12742 tablet_service.cc:1467] Processing CreateTablet for tablet c651fe3e2ce040c6b9dde98176435813 (DEFAULT_TABLE table=test-table [id=505b41bc29e84ec0885bdf6f664285f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:21.299613 12881 tablet_service.cc:1467] Processing CreateTablet for tablet c651fe3e2ce040c6b9dde98176435813 (DEFAULT_TABLE table=test-table [id=505b41bc29e84ec0885bdf6f664285f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:21.299942 12602 tablet_service.cc:1467] Processing CreateTablet for tablet c651fe3e2ce040c6b9dde98176435813 (DEFAULT_TABLE table=test-table [id=505b41bc29e84ec0885bdf6f664285f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:21.300591 12742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c651fe3e2ce040c6b9dde98176435813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:21.301304 12602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c651fe3e2ce040c6b9dde98176435813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:21.301821 12881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c651fe3e2ce040c6b9dde98176435813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:21.313087 12970 tablet_bootstrap.cc:492] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Bootstrap starting.
I20260924 19:12:21.316955 12970 tablet_bootstrap.cc:654] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:21.318418 12971 tablet_bootstrap.cc:492] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Bootstrap starting.
I20260924 19:12:21.318482 12970 log.cc:824] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:21.319851 12972 tablet_bootstrap.cc:492] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Bootstrap starting.
I20260924 19:12:21.325163 12972 tablet_bootstrap.cc:654] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:21.327050 12972 log.cc:824] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:21.329135 12971 tablet_bootstrap.cc:654] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:21.330435 12970 tablet_bootstrap.cc:492] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: No bootstrap required, opened a new log
I20260924 19:12:21.330757 12970 ts_tablet_manager.cc:1397] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260924 19:12:21.331136 12971 log.cc:824] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:21.331235 12972 tablet_bootstrap.cc:492] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: No bootstrap required, opened a new log
I20260924 19:12:21.331502 12972 ts_tablet_manager.cc:1397] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 19:12:21.334683 12971 tablet_bootstrap.cc:492] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: No bootstrap required, opened a new log
I20260924 19:12:21.335035 12971 ts_tablet_manager.cc:1397] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20260924 19:12:21.340317 12970 raft_consensus.cc:348] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.340730 12970 raft_consensus.cc:374] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:21.340883 12970 raft_consensus.cc:729] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d377fe07fc34749befba9ef02d3292f, State: Initialized, Role: FOLLOWER
I20260924 19:12:21.341572 12970 consensus_queue.cc:260] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.342681 12812 heartbeater.cc:503] Master 127.11.18.126:37703 was elected leader, sending a full tablet report...
I20260924 19:12:21.343379 12970 ts_tablet_manager.cc:1428] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260924 19:12:21.343276 12972 raft_consensus.cc:348] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.343777 12972 raft_consensus.cc:374] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:21.343889 12972 raft_consensus.cc:729] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f6cf52f8c44713a7057ceb5fc29d57, State: Initialized, Role: FOLLOWER
I20260924 19:12:21.344697 12972 consensus_queue.cc:260] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.346025 12951 heartbeater.cc:503] Master 127.11.18.126:37703 was elected leader, sending a full tablet report...
I20260924 19:12:21.347221 12972 ts_tablet_manager.cc:1428] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260924 19:12:21.347389 12971 raft_consensus.cc:348] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.347882 12971 raft_consensus.cc:374] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:21.348021 12971 raft_consensus.cc:729] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba458d46ffe4bca9a95a75dd13ee0a8, State: Initialized, Role: FOLLOWER
I20260924 19:12:21.348768 12971 consensus_queue.cc:260] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.351534 12971 ts_tablet_manager.cc:1428] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260924 19:12:21.382962 12813 tablet.cc:2369] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:12:21.431640 12952 tablet.cc:2369] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:12:21.583210 12673 tablet.cc:2369] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:12:21.685875 12976 raft_consensus.cc:482] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:21.686219 12976 raft_consensus.cc:504] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.687984 12976 leader_election.cc:288] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017), eba458d46ffe4bca9a95a75dd13ee0a8 (127.11.18.65:41843)
I20260924 19:12:21.694911 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.688375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:49895 (local address 127.11.18.67:40017)
0924 19:12:21.688568 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.688577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.689535 (+ 958us) server_negotiation.cc:407] Connection header received
0924 19:12:21.689786 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.689790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.689849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.689948 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:21.690627 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.691322 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.693125 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.693947 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.694136 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.694216 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.694498 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.694630 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:12:21.695127 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.688382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:40017 (local address 127.11.18.66:49895)
0924 19:12:21.689416 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:21.689442 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:21.689616 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:21.689974 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:21.689978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:21.690004 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:21.690468 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.690479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.691457 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.691462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.692975 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.692986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.694186 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.694194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.694349 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.694372 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 19:12:21.694473 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:12:21.694601 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":33,"spinlock_wait_cycles":29056,"thread_start_us":430,"threads_started":1}
I20260924 19:12:21.695305 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.688511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:44269 (local address 127.11.18.65:41843)
0924 19:12:21.688717 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.688725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.689701 (+ 976us) server_negotiation.cc:407] Connection header received
0924 19:12:21.689893 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.689898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.689981 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.690083 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:21.690780 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.691471 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.693164 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.694000 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.694186 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.694263 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.694949 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.695074 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:12:21.695454 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.688514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:41843 (local address 127.11.18.66:44269)
0924 19:12:21.689588 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:21.689604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:21.689757 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:21.690178 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:21.690184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:21.690233 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:21.690616 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.690625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.691600 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.691608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.693008 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:21.693019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.694232 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.694239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:21.694785 (+ 546us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.694811 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 19:12:21.694912 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:12:21.695033 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":23,"thread_start_us":528,"threads_started":1}
I20260924 19:12:21.696255 12901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651fe3e2ce040c6b9dde98176435813" candidate_uuid: "6d377fe07fc34749befba9ef02d3292f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" is_pre_election: true
I20260924 19:12:21.696343 12622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651fe3e2ce040c6b9dde98176435813" candidate_uuid: "6d377fe07fc34749befba9ef02d3292f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" is_pre_election: true
I20260924 19:12:21.696774 12901 raft_consensus.cc:2454] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d377fe07fc34749befba9ef02d3292f in term 0.
I20260924 19:12:21.696926 12622 raft_consensus.cc:2454] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d377fe07fc34749befba9ef02d3292f in term 0.
I20260924 19:12:21.697454 12693 leader_election.cc:302] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d377fe07fc34749befba9ef02d3292f, c8f6cf52f8c44713a7057ceb5fc29d57; no voters:
I20260924 19:12:21.697903 12976 raft_consensus.cc:2781] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:21.698057 12976 raft_consensus.cc:482] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:21.698158 12976 raft_consensus.cc:3037] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:21.700249 12976 raft_consensus.cc:504] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.700845 12976 leader_election.cc:288] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [CANDIDATE]: Term 1 election: Requested vote from peers c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017), eba458d46ffe4bca9a95a75dd13ee0a8 (127.11.18.65:41843)
I20260924 19:12:21.701387 12901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651fe3e2ce040c6b9dde98176435813" candidate_uuid: "6d377fe07fc34749befba9ef02d3292f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57"
I20260924 19:12:21.701472 12622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651fe3e2ce040c6b9dde98176435813" candidate_uuid: "6d377fe07fc34749befba9ef02d3292f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8"
I20260924 19:12:21.701596 12901 raft_consensus.cc:3037] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:21.701745 12622 raft_consensus.cc:3037] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:21.703925 12901 raft_consensus.cc:2454] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d377fe07fc34749befba9ef02d3292f in term 1.
I20260924 19:12:21.704371 12693 leader_election.cc:302] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d377fe07fc34749befba9ef02d3292f, c8f6cf52f8c44713a7057ceb5fc29d57; no voters:
I20260924 19:12:21.704378 12622 raft_consensus.cc:2454] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d377fe07fc34749befba9ef02d3292f in term 1.
I20260924 19:12:21.704658 12976 raft_consensus.cc:2781] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:21.705096 12976 raft_consensus.cc:686] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 1 LEADER]: Becoming Leader. State: Replica: 6d377fe07fc34749befba9ef02d3292f, State: Running, Role: LEADER
I20260924 19:12:21.705633 12976 consensus_queue.cc:237] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } }
I20260924 19:12:21.710332 12465 catalog_manager.cc:5495] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d377fe07fc34749befba9ef02d3292f (127.11.18.66). New cstate: current_term: 1 leader_uuid: "6d377fe07fc34749befba9ef02d3292f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } health_report { overall_health: UNKNOWN } } }
I20260924 19:12:21.747823 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:21.740696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.11.18.66:45469)
0924 19:12:21.740998 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:21.741004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:21.741449 (+ 445us) server_negotiation.cc:407] Connection header received
0924 19:12:21.741765 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:21.741771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:21.741870 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:21.742006 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:21.742648 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.743504 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.745490 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:21.745992 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:21.746089 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:21.746648 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:21.747183 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:21.747292 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:21.747442 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:12:21.747507 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:12:21.756325 12901 raft_consensus.cc:1261] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Refusing update from remote peer 6d377fe07fc34749befba9ef02d3292f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:21.756403 12622 raft_consensus.cc:1261] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Refusing update from remote peer 6d377fe07fc34749befba9ef02d3292f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:21.757375 12981 consensus_queue.cc:1035] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:21.757742 12976 consensus_queue.cc:1035] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:21.769747 12986 mvcc.cc:205] Tried to move back new op lower bound from 7332975172624269312 to 7332975172430180352. Current Snapshot: MvccSnapshot[applied={T|T < 7332975172624269312}]
I20260924 19:12:21.790469 12465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:21.790778 12465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:21.794916 12465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7: Sending DeleteTablet for 3 replicas of tablet c651fe3e2ce040c6b9dde98176435813
I20260924 19:12:21.796522 12602 tablet_service.cc:1514] Processing DeleteTablet for tablet c651fe3e2ce040c6b9dde98176435813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:21 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:55494
I20260924 19:12:21.796522 12881 tablet_service.cc:1514] Processing DeleteTablet for tablet c651fe3e2ce040c6b9dde98176435813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:21 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:53566
I20260924 19:12:21.797067 12742 tablet_service.cc:1514] Processing DeleteTablet for tablet c651fe3e2ce040c6b9dde98176435813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:21 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:34530
I20260924 19:12:21.803578 12465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:12:21.804512 12465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:21.806865 12997 tablet_replica.cc:330] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: stopping tablet replica
I20260924 19:12:21.807188 12996 tablet_replica.cc:330] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: stopping tablet replica
I20260924 19:12:21.807457 12997 raft_consensus.cc:2229] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:12:21.807828 12996 raft_consensus.cc:2229] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:21.808149 12997 raft_consensus.cc:2258] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:21.808442 12996 raft_consensus.cc:2258] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:21.810693 12996 ts_tablet_manager.cc:1905] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:21.816133 12742 tablet_service.cc:1467] Processing CreateTablet for tablet 2dd47166220b4a81a9bf72fa63dd7405 (DEFAULT_TABLE table=test-table [id=18f34503b2e5486dbd99aee6e48b2429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:21.816808 12742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dd47166220b4a81a9bf72fa63dd7405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:21.818322 12996 ts_tablet_manager.cc:1918] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:12:21.818604 12996 log.cc:1196] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/c651fe3e2ce040c6b9dde98176435813
I20260924 19:12:21.819233 12996 ts_tablet_manager.cc:1939] T c651fe3e2ce040c6b9dde98176435813 P eba458d46ffe4bca9a95a75dd13ee0a8: Deleting consensus metadata
I20260924 19:12:21.821571 12452 catalog_manager.cc:4841] TS eba458d46ffe4bca9a95a75dd13ee0a8 (127.11.18.65:41843): tablet c651fe3e2ce040c6b9dde98176435813 (table test-table [id=505b41bc29e84ec0885bdf6f664285f8]) successfully deleted
I20260924 19:12:21.821871 12880 tablet_service.cc:1467] Processing CreateTablet for tablet 2dd47166220b4a81a9bf72fa63dd7405 (DEFAULT_TABLE table=test-table [id=18f34503b2e5486dbd99aee6e48b2429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:21.822542 12880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dd47166220b4a81a9bf72fa63dd7405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:21.822623 12995 tablet_replica.cc:330] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: stopping tablet replica
I20260924 19:12:21.822507 12602 tablet_service.cc:1467] Processing CreateTablet for tablet 2dd47166220b4a81a9bf72fa63dd7405 (DEFAULT_TABLE table=test-table [id=18f34503b2e5486dbd99aee6e48b2429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:21.823084 12602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dd47166220b4a81a9bf72fa63dd7405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:21.823129 12995 raft_consensus.cc:2229] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:21.823679 12995 raft_consensus.cc:2258] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:21.829816 12972 tablet_bootstrap.cc:492] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: Bootstrap starting.
I20260924 19:12:21.829818 12971 tablet_bootstrap.cc:492] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: Bootstrap starting.
I20260924 19:12:21.831468 12971 tablet_bootstrap.cc:654] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:21.831480 12972 tablet_bootstrap.cc:654] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:21.832856 12997 ts_tablet_manager.cc:1905] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:21.835748 12995 ts_tablet_manager.cc:1905] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:21.838829 12970 tablet_bootstrap.cc:492] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: Bootstrap starting.
I20260924 19:12:21.841275 12970 tablet_bootstrap.cc:654] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:21.844599 12971 tablet_bootstrap.cc:492] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: No bootstrap required, opened a new log
I20260924 19:12:21.844834 12971 ts_tablet_manager.cc:1397] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260924 19:12:21.845921 12971 raft_consensus.cc:348] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:21.846196 12971 raft_consensus.cc:374] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:21.846364 12971 raft_consensus.cc:729] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba458d46ffe4bca9a95a75dd13ee0a8, State: Initialized, Role: FOLLOWER
I20260924 19:12:21.846678 12971 consensus_queue.cc:260] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:21.847447 12971 ts_tablet_manager.cc:1428] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:12:21.849289 12997 ts_tablet_manager.cc:1918] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:12:21.849501 12997 log.cc:1196] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/c651fe3e2ce040c6b9dde98176435813
I20260924 19:12:21.849606 12972 tablet_bootstrap.cc:492] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: No bootstrap required, opened a new log
I20260924 19:12:21.849812 12972 ts_tablet_manager.cc:1397] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.013s
I20260924 19:12:21.850317 12997 ts_tablet_manager.cc:1939] T c651fe3e2ce040c6b9dde98176435813 P 6d377fe07fc34749befba9ef02d3292f: Deleting consensus metadata
I20260924 19:12:21.850874 12972 raft_consensus.cc:348] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:21.851169 12972 raft_consensus.cc:374] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:21.851294 12972 raft_consensus.cc:729] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f6cf52f8c44713a7057ceb5fc29d57, State: Initialized, Role: FOLLOWER
I20260924 19:12:21.851611 12972 consensus_queue.cc:260] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:21.853971 12451 catalog_manager.cc:4841] TS 6d377fe07fc34749befba9ef02d3292f (127.11.18.66:45469): tablet c651fe3e2ce040c6b9dde98176435813 (table test-table [id=505b41bc29e84ec0885bdf6f664285f8]) successfully deleted
I20260924 19:12:21.854730 12972 ts_tablet_manager.cc:1428] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 19:12:21.855758 12970 tablet_bootstrap.cc:492] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: No bootstrap required, opened a new log
I20260924 19:12:21.855955 12970 ts_tablet_manager.cc:1397] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20260924 19:12:21.856861 12995 ts_tablet_manager.cc:1918] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:12:21.857043 12995 log.cc:1196] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/c651fe3e2ce040c6b9dde98176435813
I20260924 19:12:21.856980 12970 raft_consensus.cc:348] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:21.857239 12970 raft_consensus.cc:374] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:21.857419 12970 raft_consensus.cc:729] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d377fe07fc34749befba9ef02d3292f, State: Initialized, Role: FOLLOWER
I20260924 19:12:21.857672 12995 ts_tablet_manager.cc:1939] T c651fe3e2ce040c6b9dde98176435813 P c8f6cf52f8c44713a7057ceb5fc29d57: Deleting consensus metadata
I20260924 19:12:21.857762 12970 consensus_queue.cc:260] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:21.858472 12970 ts_tablet_manager.cc:1428] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:12:21.859841 12450 catalog_manager.cc:4841] TS c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017): tablet c651fe3e2ce040c6b9dde98176435813 (table test-table [id=505b41bc29e84ec0885bdf6f664285f8]) successfully deleted
I20260924 19:12:22.131485 12978 raft_consensus.cc:482] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:22.131783 12978 raft_consensus.cc:504] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.133533 12978 leader_election.cc:288] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d377fe07fc34749befba9ef02d3292f (127.11.18.66:45469), c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017)
I20260924 19:12:22.143360 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.134478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:40445 (local address 127.11.18.66:45469)
0924 19:12:22.134691 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.134697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.134718 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:12:22.134810 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.134816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.134939 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.135051 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:22.135833 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.137146 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.140664 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.142157 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.142345 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.142427 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.142888 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.143041 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:12:22.143450 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.133735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:45469 (local address 127.11.18.65:40445)
0924 19:12:22.134371 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:22.134397 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:22.134580 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:22.135083 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:22.135089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:22.135123 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:22.135517 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.135527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.137296 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.137304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.140443 (+ 3139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.140461 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.142428 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.142437 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.142685 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.142724 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:12:22.142859 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 19:12:22.143025 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":161,"thread_start_us":1457,"threads_started":1}
I20260924 19:12:22.143906 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.134112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:42739 (local address 127.11.18.67:40017)
0924 19:12:22.134310 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.134317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.135177 (+ 860us) server_negotiation.cc:407] Connection header received
0924 19:12:22.135504 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.135510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.135611 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.135713 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:22.136479 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.137579 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.141501 (+ 3922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.142476 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.142653 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.142722 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.143224 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.143377 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:12:22.144527 12766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dd47166220b4a81a9bf72fa63dd7405" candidate_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d377fe07fc34749befba9ef02d3292f" is_pre_election: true
I20260924 19:12:22.144868 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.133836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:40017 (local address 127.11.18.65:42739)
0924 19:12:22.135075 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:22.135101 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:22.135292 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:22.135875 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:22.135881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:22.135908 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:22.136277 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.136292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.137996 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.138001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.140444 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.140461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.142786 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.142793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.142998 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.143037 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:12:22.144250 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0924 19:12:22.144415 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":82,"thread_start_us":967,"threads_started":1}
I20260924 19:12:22.145092 12766 raft_consensus.cc:2454] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eba458d46ffe4bca9a95a75dd13ee0a8 in term 0.
I20260924 19:12:22.145541 12901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dd47166220b4a81a9bf72fa63dd7405" candidate_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" is_pre_election: true
I20260924 19:12:22.145810 12901 raft_consensus.cc:2454] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eba458d46ffe4bca9a95a75dd13ee0a8 in term 0.
I20260924 19:12:22.146396 12553 leader_election.cc:302] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d377fe07fc34749befba9ef02d3292f, c8f6cf52f8c44713a7057ceb5fc29d57, eba458d46ffe4bca9a95a75dd13ee0a8; no voters:
I20260924 19:12:22.146903 12978 raft_consensus.cc:2781] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:22.147047 12978 raft_consensus.cc:482] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:22.147167 12978 raft_consensus.cc:3037] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:22.149411 12978 raft_consensus.cc:504] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.150177 12978 leader_election.cc:288] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [CANDIDATE]: Term 1 election: Requested vote from peers 6d377fe07fc34749befba9ef02d3292f (127.11.18.66:45469), c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017)
I20260924 19:12:22.150703 12766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dd47166220b4a81a9bf72fa63dd7405" candidate_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d377fe07fc34749befba9ef02d3292f"
I20260924 19:12:22.150965 12766 raft_consensus.cc:3037] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:22.151046 12901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dd47166220b4a81a9bf72fa63dd7405" candidate_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57"
I20260924 19:12:22.151286 12901 raft_consensus.cc:3037] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:22.153920 12766 raft_consensus.cc:2454] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba458d46ffe4bca9a95a75dd13ee0a8 in term 1.
I20260924 19:12:22.154362 12901 raft_consensus.cc:2454] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba458d46ffe4bca9a95a75dd13ee0a8 in term 1.
I20260924 19:12:22.154397 12554 leader_election.cc:302] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d377fe07fc34749befba9ef02d3292f, eba458d46ffe4bca9a95a75dd13ee0a8; no voters:
I20260924 19:12:22.154918 12978 raft_consensus.cc:2781] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:22.155481 12978 raft_consensus.cc:686] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 LEADER]: Becoming Leader. State: Replica: eba458d46ffe4bca9a95a75dd13ee0a8, State: Running, Role: LEADER
I20260924 19:12:22.155905 12978 consensus_queue.cc:237] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.159689 12465 catalog_manager.cc:5495] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to eba458d46ffe4bca9a95a75dd13ee0a8 (127.11.18.65). New cstate: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } health_report { overall_health: UNKNOWN } } }
Sep 24 19:12:22 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277139, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:22.250587 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.232218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.11.18.126:37703)
0924 19:12:22.233477 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.233485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.233754 (+ 269us) server_negotiation.cc:407] Connection header received
0924 19:12:22.233982 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.233989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.234060 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.234182 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:22.235436 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.236297 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.237693 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.238189 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.238336 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.242412 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:22.242418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:22.242421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:22.242461 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:22.246423 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.247385 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.247390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.247395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.247550 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.247955 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.247960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.247964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.248293 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:22.248422 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.250000 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.250208 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043,"thread_start_us":878,"threads_started":1}
I20260924 19:12:22.251128 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.232154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37703 (local address 127.0.0.1:44448)
0924 19:12:22.233630 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:22.233647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:22.233843 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:22.234205 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:22.234210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:22.234986 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:22.235244 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.235256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.236457 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.236464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.237537 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.237549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.238379 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.238386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.238563 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.239346 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:22.239366 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:22.242154 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:22.246611 (+ 4457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:22.246627 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:22.246634 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:22.247219 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:22.247668 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:22.247672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:22.247677 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:22.247851 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:22.248500 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:22.248510 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:22.248686 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 19:12:22.249269 (+ 583us) client_negotiation.cc:240] Negotiation successful
0924 19:12:22.249609 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"thread_start_us":1193,"threads_started":1}
I20260924 19:12:22.264732 12465 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:44448:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:12:22.265900 12465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:22.274255 12742 tablet_service.cc:1467] Processing CreateTablet for tablet 413155c132b24b4e83e2170f50b9153f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65da7f306e204aee80559599803a3a64]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:12:22.274267 12602 tablet_service.cc:1467] Processing CreateTablet for tablet 413155c132b24b4e83e2170f50b9153f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65da7f306e204aee80559599803a3a64]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:12:22.274921 12742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413155c132b24b4e83e2170f50b9153f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:22.274924 12602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413155c132b24b4e83e2170f50b9153f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:22.275601 12880 tablet_service.cc:1467] Processing CreateTablet for tablet 413155c132b24b4e83e2170f50b9153f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65da7f306e204aee80559599803a3a64]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:12:22.276137 12880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413155c132b24b4e83e2170f50b9153f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:22.281302 12970 tablet_bootstrap.cc:492] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: Bootstrap starting.
I20260924 19:12:22.282462 12971 tablet_bootstrap.cc:492] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: Bootstrap starting.
I20260924 19:12:22.283119 12972 tablet_bootstrap.cc:492] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: Bootstrap starting.
I20260924 19:12:22.283205 12970 tablet_bootstrap.cc:654] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:22.284428 12971 tablet_bootstrap.cc:654] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:22.284793 12972 tablet_bootstrap.cc:654] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:22.286031 12970 tablet_bootstrap.cc:492] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: No bootstrap required, opened a new log
I20260924 19:12:22.286324 12970 ts_tablet_manager.cc:1397] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 19:12:22.287218 12972 tablet_bootstrap.cc:492] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: No bootstrap required, opened a new log
I20260924 19:12:22.287132 12970 raft_consensus.cc:348] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.287400 12972 ts_tablet_manager.cc:1397] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 19:12:22.287438 12970 raft_consensus.cc:374] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:22.287541 12970 raft_consensus.cc:729] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d377fe07fc34749befba9ef02d3292f, State: Initialized, Role: FOLLOWER
I20260924 19:12:22.287912 12971 tablet_bootstrap.cc:492] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: No bootstrap required, opened a new log
I20260924 19:12:22.287848 12970 consensus_queue.cc:260] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.288151 12971 ts_tablet_manager.cc:1397] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260924 19:12:22.288184 12972 raft_consensus.cc:348] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.288437 12972 raft_consensus.cc:374] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:22.288538 12972 raft_consensus.cc:729] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f6cf52f8c44713a7057ceb5fc29d57, State: Initialized, Role: FOLLOWER
I20260924 19:12:22.288384 12990 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.288713 12990 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:22.288832 12972 consensus_queue.cc:260] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.288968 12971 raft_consensus.cc:348] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.289218 12972 ts_tablet_manager.cc:1428] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 19:12:22.289214 12971 raft_consensus.cc:374] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:22.289290 12971 raft_consensus.cc:729] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba458d46ffe4bca9a95a75dd13ee0a8, State: Initialized, Role: FOLLOWER
I20260924 19:12:22.289564 12971 consensus_queue.cc:260] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.289665 12977 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.290011 12977 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:22.290012 12978 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.290202 12971 ts_tablet_manager.cc:1428] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:12:22.290298 12978 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:22.291108 12970 ts_tablet_manager.cc:1428] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:12:22.377261 12978 raft_consensus.cc:482] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:22.377519 12978 raft_consensus.cc:504] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.378217 12978 leader_election.cc:288] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d377fe07fc34749befba9ef02d3292f (127.11.18.66:45469), c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017)
I20260924 19:12:22.378863 12766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413155c132b24b4e83e2170f50b9153f" candidate_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d377fe07fc34749befba9ef02d3292f" is_pre_election: true
I20260924 19:12:22.378863 12901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413155c132b24b4e83e2170f50b9153f" candidate_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" is_pre_election: true
I20260924 19:12:22.379170 12901 raft_consensus.cc:2454] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eba458d46ffe4bca9a95a75dd13ee0a8 in term 0.
I20260924 19:12:22.379176 12766 raft_consensus.cc:2454] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eba458d46ffe4bca9a95a75dd13ee0a8 in term 0.
I20260924 19:12:22.379637 12553 leader_election.cc:302] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8f6cf52f8c44713a7057ceb5fc29d57, eba458d46ffe4bca9a95a75dd13ee0a8; no voters:
I20260924 19:12:22.380007 12978 raft_consensus.cc:2781] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:22.380147 12978 raft_consensus.cc:482] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:22.380303 12978 raft_consensus.cc:3037] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:22.382920 12978 raft_consensus.cc:504] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.383641 12978 leader_election.cc:288] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [CANDIDATE]: Term 1 election: Requested vote from peers 6d377fe07fc34749befba9ef02d3292f (127.11.18.66:45469), c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017)
I20260924 19:12:22.384274 12901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413155c132b24b4e83e2170f50b9153f" candidate_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57"
I20260924 19:12:22.384271 12766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413155c132b24b4e83e2170f50b9153f" candidate_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d377fe07fc34749befba9ef02d3292f"
I20260924 19:12:22.384526 12766 raft_consensus.cc:3037] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:22.384546 12901 raft_consensus.cc:3037] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:22.387486 12901 raft_consensus.cc:2454] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba458d46ffe4bca9a95a75dd13ee0a8 in term 1.
I20260924 19:12:22.387486 12766 raft_consensus.cc:2454] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba458d46ffe4bca9a95a75dd13ee0a8 in term 1.
I20260924 19:12:22.388013 12554 leader_election.cc:302] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d377fe07fc34749befba9ef02d3292f, c8f6cf52f8c44713a7057ceb5fc29d57, eba458d46ffe4bca9a95a75dd13ee0a8; no voters:
I20260924 19:12:22.388484 12978 raft_consensus.cc:2781] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:22.388742 12978 raft_consensus.cc:686] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 LEADER]: Becoming Leader. State: Replica: eba458d46ffe4bca9a95a75dd13ee0a8, State: Running, Role: LEADER
I20260924 19:12:22.389026 12978 consensus_queue.cc:237] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } }
I20260924 19:12:22.390805 13003 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: TxnStatusTablet state changed. Reason: New leader eba458d46ffe4bca9a95a75dd13ee0a8. Latest consensus state: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.391187 13003 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:12:22.391971 13016 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:12:22.392230 13016 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:12:22.392731 12464 catalog_manager.cc:5495] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to eba458d46ffe4bca9a95a75dd13ee0a8 (127.11.18.65). New cstate: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } health_report { overall_health: UNKNOWN } } }
I20260924 19:12:22.429364 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.414733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:41843 (local address 127.0.0.1:55506)
0924 19:12:22.415015 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:22.415041 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:22.415207 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:22.422124 (+ 6917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:22.422132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:22.422155 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:12:22.422388 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.422401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.424346 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.424354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.426253 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.426272 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.427138 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.427145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.427369 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.427559 (+ 190us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:12:22.428644 (+ 1085us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:12:22.428672 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 19:12:22.428779 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:12:22.428962 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 19:12:22.429546 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.414735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.11.18.65:41843)
0924 19:12:22.421624 (+ 6889us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.421629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.421652 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:12:22.421753 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.421756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.421858 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.422041 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:22.422591 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.424194 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.426467 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.426952 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.427058 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.427845 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:22.428514 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:22.428643 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.428826 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.428903 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6762,"thread_start_us":6572,"threads_started":1}
I20260924 19:12:22.435258 12901 raft_consensus.cc:1261] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Refusing update from remote peer eba458d46ffe4bca9a95a75dd13ee0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:22.435488 12766 raft_consensus.cc:1261] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Refusing update from remote peer eba458d46ffe4bca9a95a75dd13ee0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:22.436067 13003 consensus_queue.cc:1035] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:22.436430 12978 consensus_queue.cc:1035] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:22.443094 12977 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: TxnStatusTablet state changed. Reason: New leader eba458d46ffe4bca9a95a75dd13ee0a8. Latest consensus state: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.443503 12977 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:22.447290 12978 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.447659 12978 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:12:22.452556 12990 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: TxnStatusTablet state changed. Reason: New leader eba458d46ffe4bca9a95a75dd13ee0a8. Latest consensus state: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.452906 12990 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:22.458945 13003 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.461304 13003 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P eba458d46ffe4bca9a95a75dd13ee0a8: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:12:22.461757 13022 mvcc.cc:205] Tried to move back new op lower bound from 7332975175406325760 to 7332975175228268544. Current Snapshot: MvccSnapshot[applied={T|T < 7332975175406325760}]
I20260924 19:12:22.468681 12977 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.468966 12977 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P c8f6cf52f8c44713a7057ceb5fc29d57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:22.474506 12990 tablet_replica.cc:439] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba458d46ffe4bca9a95a75dd13ee0a8" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41843 } } peers { permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 } } peers { permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 } } }
I20260924 19:12:22.474888 12990 tablet_replica.cc:442] T 413155c132b24b4e83e2170f50b9153f P 6d377fe07fc34749befba9ef02d3292f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:22.488588 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.478605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.11.18.65:41843)
0924 19:12:22.478855 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.478865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.482750 (+ 3885us) server_negotiation.cc:407] Connection header received
0924 19:12:22.482931 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.482935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.483002 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.483084 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:22.483614 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.484449 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.486558 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.487065 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.487176 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.487686 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:22.487798 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:22.487881 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.488081 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.488153 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:12:22.508285 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.501185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:44675 (local address 127.11.18.65:41843)
0924 19:12:22.501432 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.501438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.502124 (+ 686us) server_negotiation.cc:407] Connection header received
0924 19:12:22.502272 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.502279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.502425 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.502517 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:22.503073 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.504301 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.506093 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.506788 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.506917 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.507436 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:22.507564 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:22.507663 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.507772 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.507854 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":62}
I20260924 19:12:22.508309 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.501135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:41843 (local address 127.11.18.65:44675)
0924 19:12:22.502015 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:22.502033 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:22.502171 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:22.502528 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:22.502531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:22.502541 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:12:22.502891 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.502898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.504456 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.504461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.505955 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:22.505967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.506986 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.506992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:22.507178 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.507255 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:12:22.507672 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:12:22.507680 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:12:22.507745 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:12:22.507859 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":4,"thread_start_us":503,"threads_started":1}
I20260924 19:12:22.518419 12901 raft_consensus.cc:1261] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Refusing update from remote peer eba458d46ffe4bca9a95a75dd13ee0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:22.518965 12761 raft_consensus.cc:1261] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Refusing update from remote peer eba458d46ffe4bca9a95a75dd13ee0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:22.518963 13028 consensus_queue.cc:1035] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f6cf52f8c44713a7057ceb5fc29d57" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:22.519449 12978 consensus_queue.cc:1035] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d377fe07fc34749befba9ef02d3292f" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 45469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:22.527308 13020 mvcc.cc:205] Tried to move back new op lower bound from 7332975175750684672 to 7332975174274670592. Current Snapshot: MvccSnapshot[applied={T|T < 7332975175750684672}]
I20260924 19:12:22.584774 12463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:22.585058 12463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:22.589068 12463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa5a50820cd843aaa91acb1f0decffc7: Sending DeleteTablet for 3 replicas of tablet 2dd47166220b4a81a9bf72fa63dd7405
I20260924 19:12:22.590091 12742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dd47166220b4a81a9bf72fa63dd7405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:22 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:34530
I20260924 19:12:22.590479 12880 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dd47166220b4a81a9bf72fa63dd7405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:22 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:53566
I20260924 19:12:22.590495 12601 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dd47166220b4a81a9bf72fa63dd7405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:22 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:55494
I20260924 19:12:22.601068 13037 tablet_replica.cc:330] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: stopping tablet replica
I20260924 19:12:22.601686 13037 raft_consensus.cc:2229] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:22.601716 13039 tablet_replica.cc:330] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: stopping tablet replica
I20260924 19:12:22.601970 13039 raft_consensus.cc:2229] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:22.602083 13037 raft_consensus.cc:2258] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:22.602479 13039 raft_consensus.cc:2258] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:22.603029 13038 tablet_replica.cc:330] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: stopping tablet replica
I20260924 19:12:22.603856 13039 ts_tablet_manager.cc:1905] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:22.604175 13038 raft_consensus.cc:2229] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:12:22.604458 13037 ts_tablet_manager.cc:1905] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:22.604880 13038 raft_consensus.cc:2258] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:22.606034 13038 ts_tablet_manager.cc:1905] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:22.611472 13037 ts_tablet_manager.cc:1918] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:22.611657 13039 ts_tablet_manager.cc:1918] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:22.611847 13039 log.cc:1196] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/2dd47166220b4a81a9bf72fa63dd7405
I20260924 19:12:22.612416 13039 ts_tablet_manager.cc:1939] T 2dd47166220b4a81a9bf72fa63dd7405 P c8f6cf52f8c44713a7057ceb5fc29d57: Deleting consensus metadata
I20260924 19:12:22.613685 12450 catalog_manager.cc:4841] TS c8f6cf52f8c44713a7057ceb5fc29d57 (127.11.18.67:40017): tablet 2dd47166220b4a81a9bf72fa63dd7405 (table test-table [id=18f34503b2e5486dbd99aee6e48b2429]) successfully deleted
I20260924 19:12:22.614249 13038 ts_tablet_manager.cc:1918] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:22.614385 13038 log.cc:1196] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/2dd47166220b4a81a9bf72fa63dd7405
I20260924 19:12:22.615063 13037 log.cc:1196] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/2dd47166220b4a81a9bf72fa63dd7405
I20260924 19:12:22.615222 13038 ts_tablet_manager.cc:1939] T 2dd47166220b4a81a9bf72fa63dd7405 P eba458d46ffe4bca9a95a75dd13ee0a8: Deleting consensus metadata
I20260924 19:12:22.615563 13037 ts_tablet_manager.cc:1939] T 2dd47166220b4a81a9bf72fa63dd7405 P 6d377fe07fc34749befba9ef02d3292f: Deleting consensus metadata
I20260924 19:12:22.616429 12452 catalog_manager.cc:4841] TS eba458d46ffe4bca9a95a75dd13ee0a8 (127.11.18.65:41843): tablet 2dd47166220b4a81a9bf72fa63dd7405 (table test-table [id=18f34503b2e5486dbd99aee6e48b2429]) successfully deleted
I20260924 19:12:22.616698 12451 catalog_manager.cc:4841] TS 6d377fe07fc34749befba9ef02d3292f (127.11.18.66:45469): tablet 2dd47166220b4a81a9bf72fa63dd7405 (table test-table [id=18f34503b2e5486dbd99aee6e48b2429]) successfully deleted
Sep 24 19:12:22 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:22.631021 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.603713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.11.18.65:41843)
0924 19:12:22.603977 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.603984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.612291 (+ 8307us) server_negotiation.cc:407] Connection header received
0924 19:12:22.612648 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.612654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.612750 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.612827 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:22.614975 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.616220 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.617840 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.618672 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.618819 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.623478 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:22.623490 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:22.623509 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:22.623566 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:22.627689 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.628602 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.628613 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.628617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.628782 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.629194 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.629199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.629203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.629761 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:22.630108 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.630393 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.630613 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260924 19:12:22.631760 12667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55532
I20260924 19:12:22.653263 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.638528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.11.18.126:37703)
0924 19:12:22.638806 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.638815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.639327 (+ 512us) server_negotiation.cc:407] Connection header received
0924 19:12:22.639530 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.639536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.639627 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.639735 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:22.641081 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.641905 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.643125 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.643583 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.643724 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.646881 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:22.646895 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:22.646899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:22.646935 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:22.649780 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.650687 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.650692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.650694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.650816 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.651354 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.651363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.651365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.651839 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:22.652178 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.652487 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.652724 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260924 19:12:22.654157 12463 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44452
I20260924 19:12:22.657687 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12536
I20260924 19:12:22.678664 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.658963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.11.18.65:41843)
0924 19:12:22.659167 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.659173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.661881 (+ 2708us) server_negotiation.cc:407] Connection header received
0924 19:12:22.662107 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.662114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.662213 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.662332 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:22.664036 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.665396 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.667340 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.668115 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.668259 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.671465 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:22.671473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:22.671479 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:22.671533 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:22.675544 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.676324 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.676332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.676336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.676520 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.676969 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.676974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.676978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.677404 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:22.677694 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.677998 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.678258 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 19:12:22.679374 12667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55542
W20260924 19:12:22.680030 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12536: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:22.680238 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 12536
I20260924 19:12:22.701720 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12676
Sep 24 19:12:22 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:12:22.719265 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.702743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.11.18.66:45469)
0924 19:12:22.703291 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.703296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.703313 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:12:22.703368 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.703372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.703431 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.703528 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:22.704881 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.705926 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.707484 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.708183 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.708303 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.712871 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:22.712888 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:22.712906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:22.712957 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:22.716342 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.717096 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.717104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.717106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.717311 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.717760 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.717764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.717766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.718109 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:22.718408 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.718651 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.718835 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":328,"threads_started":1}
W20260924 19:12:22.719915 12807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34554
W20260924 19:12:22.720571 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:22.720782 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 12676
I20260924 19:12:22.742684 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12816
Sep 24 19:12:22 dist-test-slave-0lgr krb5kdc[12420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:12:22.760475 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:22.743495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.11.18.67:40017)
0924 19:12:22.744061 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:22.744065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:22.744084 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:12:22.744141 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:22.744145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:22.744275 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:22.744422 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:22.746003 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.746794 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.748234 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:22.748713 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:22.748856 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:22.753908 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:22.753923 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:22.753940 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:22.753976 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:22.756916 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.757885 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.757894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.757896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.758129 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:22.758631 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:22.758638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:22.758645 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:22.759159 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:22.759498 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:22.759869 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 19:12:22.760099 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":334,"threads_started":1}
W20260924 19:12:22.761276 12946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53568
W20260924 19:12:22.762017 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:22.762156 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 12816
I20260924 19:12:22.779647 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 12433
I20260924 19:12:22.780866 12505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:22.898731 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 12433
2026-09-24T19:12:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5265 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:23 dist-test-slave-0lgr krb5kdc[13050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:23 dist-test-slave-0lgr krb5kdc[13050](info): set up 2 sockets
Sep 24 19:12:23 dist-test-slave-0lgr krb5kdc[13050](info): commencing operation
krb5kdc: starting...
W20260924 19:12:24.057756 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:24 dist-test-slave-0lgr krb5kdc[13050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:24.261566 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:46147
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:46147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:24.586714 13063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:24.587033 13063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:24.587132 13063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:24.594789 13063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:24.594908 13063 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:12:24.594949 13063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:24.595003 13063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:24.595059 13063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:24.595121 13063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:24.605088 13063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:46147
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:46147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:24.606403 13063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:24.607625 13063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:24.620131 13068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:24.620229 13069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:24.620879 13071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:24.621198 13063 server_base.cc:1034] running on GCE node
I20260924 19:12:24.622027 13063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:24.623519 13063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:24.624769 13063 hybrid_clock.cc:648] HybridClock initialized: now 1790277144624716 us; error 43 us; skew 500 ppm
I20260924 19:12:24.625151 13063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:24 dist-test-slave-0lgr krb5kdc[13050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277144, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:24.628547 13063 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:24.630105 13063 webserver.cc:460] Webserver started at http://127.11.18.126:43977/ using document root <none> and password file <none>
I20260924 19:12:24.630824 13063 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:24.630957 13063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:24.631299 13063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:24.634367 13063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba"
format_stamp: "Formatted at 2026-09-24 19:12:24 on dist-test-slave-0lgr"
server_key: "fef42d2bfc21ad294bb6baf6b286728b"
server_key_iv: "01b498a4155b15a71bb98c88cc73712a"
server_key_version: "encryptionkey@0"
I20260924 19:12:24.635179 13063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba"
format_stamp: "Formatted at 2026-09-24 19:12:24 on dist-test-slave-0lgr"
server_key: "fef42d2bfc21ad294bb6baf6b286728b"
server_key_iv: "01b498a4155b15a71bb98c88cc73712a"
server_key_version: "encryptionkey@0"
I20260924 19:12:24.640638 13063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:12:24.644587 13078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:24.646283 13063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:24.646523 13063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba"
format_stamp: "Formatted at 2026-09-24 19:12:24 on dist-test-slave-0lgr"
server_key: "fef42d2bfc21ad294bb6baf6b286728b"
server_key_iv: "01b498a4155b15a71bb98c88cc73712a"
server_key_version: "encryptionkey@0"
I20260924 19:12:24.646791 13063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:24.678910 13063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:24.683517 13063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:24.683830 13063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:24.706660 13063 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:46147
I20260924 19:12:24.706677 13139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:46147 every 8 connection(s)
I20260924 19:12:24.708096 13063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:24.710605 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 13063
I20260924 19:12:24.710803 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:24.711592 11337 external_mini_cluster.cc:1428] Setting key d4de0701d60b8703619c90dc98ac58a1
I20260924 19:12:24.712689 13140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:24.728233 13140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: Bootstrap starting.
Sep 24 19:12:24 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:24.732141 13140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:24.733560 13140 log.cc:824] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:24.737285 13140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: No bootstrap required, opened a new log
I20260924 19:12:24.739084 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:24.713893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.11.18.126:46147)
0924 19:12:24.714905 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:24.714920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:24.715234 (+ 314us) server_negotiation.cc:407] Connection header received
0924 19:12:24.717804 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:24.717835 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:24.718497 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:24.718978 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:24.720747 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:24.722254 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:24.724028 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:24.724911 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:24.725082 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:24.729953 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:24.729964 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:24.729986 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:24.730040 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:24.734204 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:24.735192 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:24.735199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:24.735208 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:24.735397 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:24.735826 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:24.735832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:24.735838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:24.736570 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:24.736848 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:24.737268 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 19:12:24.737745 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":445,"threads_started":1}
I20260924 19:12:24.745857 13140 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46147 } }
I20260924 19:12:24.746354 13140 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:24.746475 13140 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d992880fa54e18ab2acd3ffbf0b8ba, State: Initialized, Role: FOLLOWER
I20260924 19:12:24.747455 13140 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46147 } }
I20260924 19:12:24.747757 13145 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:24.748131 13145 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46147 } }
I20260924 19:12:24.748703 13145 leader_election.cc:302] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8d992880fa54e18ab2acd3ffbf0b8ba; no voters:
I20260924 19:12:24.749481 13145 leader_election.cc:288] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260924 19:12:24.747732 13140 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:24.749725 13146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:24.749864 13146 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:24.750025 13146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:24.752506 13146 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46147 } }
I20260924 19:12:24.752815 13146 leader_election.cc:302] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8d992880fa54e18ab2acd3ffbf0b8ba; no voters:
I20260924 19:12:24.752839 13140 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:24.753041 13146 leader_election.cc:288] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:24.753082 13140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:12:24.755970 13140 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46147 } }
I20260924 19:12:24.756319 13145 raft_consensus.cc:2741] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won
I20260924 19:12:24.756417 13140 leader_election.cc:302] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8d992880fa54e18ab2acd3ffbf0b8ba; no voters:
I20260924 19:12:24.756726 13140 leader_election.cc:288] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:12:24.756879 13146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:12:24.757632 13146 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [term 2 LEADER]: Becoming Leader. State: Replica: a8d992880fa54e18ab2acd3ffbf0b8ba, State: Running, Role: LEADER
I20260924 19:12:24.758095 13146 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46147 } }
I20260924 19:12:24.759625 13140 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:24.760164 13145 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46147 } } }
I20260924 19:12:24.760257 13148 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8d992880fa54e18ab2acd3ffbf0b8ba. Latest consensus state: current_term: 2 leader_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d992880fa54e18ab2acd3ffbf0b8ba" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46147 } } }
I20260924 19:12:24.760493 13145 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:24.760567 13148 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:24.761226 13150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:24.768826 13150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:24.781049 13150 catalog_manager.cc:1347] Generated new cluster ID: 132300fcfb1d4debb9ac0f124b82e7a3
I20260924 19:12:24.781270 13150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:24.801774 13150 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:24.803155 13150 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:24.817498 13150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: Generated new TSK 0
I20260924 19:12:24.818478 13150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:12:24.869867 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46147
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:25.198860 13167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:25.199221 13167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:25.199396 13167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:25.206673 13167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:25.206847 13167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:25.206995 13167 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:12:25.207090 13167 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:12:25.207219 13167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:25.217154 13167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:46147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:25.218515 13167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:25.219861 13167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:25.233168 13173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:25.233219 13172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:25.233258 13175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:25.234237 13167 server_base.cc:1034] running on GCE node
I20260924 19:12:25.234814 13167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:25.235791 13167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:25.236995 13167 hybrid_clock.cc:648] HybridClock initialized: now 1790277145236947 us; error 33 us; skew 500 ppm
I20260924 19:12:25.237362 13167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:25 dist-test-slave-0lgr krb5kdc[13050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277145, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:25.240536 13167 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:12:25.242167 13167 webserver.cc:460] Webserver started at http://127.11.18.65:41813/ using document root <none> and password file <none>
I20260924 19:12:25.242803 13167 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:25.242901 13167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:25.243180 13167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:25.246176 13167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560"
format_stamp: "Formatted at 2026-09-24 19:12:25 on dist-test-slave-0lgr"
server_key: "f2a9446702486b230ef1d75da094f73a"
server_key_iv: "ed7ba327acb3adf27477f0e33d30e608"
server_key_version: "encryptionkey@0"
I20260924 19:12:25.246953 13167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560"
format_stamp: "Formatted at 2026-09-24 19:12:25 on dist-test-slave-0lgr"
server_key: "f2a9446702486b230ef1d75da094f73a"
server_key_iv: "ed7ba327acb3adf27477f0e33d30e608"
server_key_version: "encryptionkey@0"
I20260924 19:12:25.252461 13167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:12:25.256134 13182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:25.257628 13167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:25.257810 13167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560"
format_stamp: "Formatted at 2026-09-24 19:12:25 on dist-test-slave-0lgr"
server_key: "f2a9446702486b230ef1d75da094f73a"
server_key_iv: "ed7ba327acb3adf27477f0e33d30e608"
server_key_version: "encryptionkey@0"
I20260924 19:12:25.258009 13167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:25.277661 13167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:25.282244 13167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:25.282545 13167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:25.285616 13167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:25.285763 13167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:25.285893 13167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:25.285985 13167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:12:25 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277145, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:25.316038 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:25.286445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:51035 (local address 127.11.18.126:46147)
0924 19:12:25.286959 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:25.286967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:25.288111 (+ 1144us) server_negotiation.cc:407] Connection header received
0924 19:12:25.291198 (+ 3087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:25.291206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:25.291309 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:25.291441 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:25.294007 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.295280 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.300109 (+ 4829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.300872 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.301025 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:25.306789 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:25.306798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:25.306804 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:25.306862 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:25.311135 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:25.312686 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:25.312693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:25.312697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:25.312863 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:25.313541 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:25.313547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:25.313551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:25.313869 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:25.314083 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:25.315124 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0924 19:12:25.315439 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":261,"threads_started":1}
I20260924 19:12:25.316702 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:25.287020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46147 (local address 127.11.18.65:51035)
0924 19:12:25.287909 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:25.287966 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:25.290853 (+ 2887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:25.291752 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:25.291768 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:25.292664 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:25.293714 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:25.293747 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.295589 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.295598 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:25.299867 (+ 4269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:25.299890 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.301163 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.301170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:25.301472 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:25.302358 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:25.302386 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:25.306402 (+ 4016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:25.311456 (+ 5054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:25.311470 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:25.311497 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:25.312460 (+ 963us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:25.313022 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:25.313034 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:25.313039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:25.313373 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:25.314192 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:25.314208 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:25.314676 (+ 468us) client_negotiation.cc:769] Sending connection context
0924 19:12:25.314930 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 19:12:25.315387 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":335,"threads_started":1}
I20260924 19:12:25.340687 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:25.323951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46147 (local address 127.11.18.65:39645)
0924 19:12:25.326128 (+ 2177us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:25.326152 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:25.326360 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:25.326771 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:25.326776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:25.327314 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:25.327625 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:25.327635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.328702 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.328708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:25.330220 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:25.330232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.331089 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.331098 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:25.331298 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:25.332296 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:25.332313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:25.333104 (+ 791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:25.337163 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:25.337171 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:25.337175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:25.337990 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:25.338495 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:25.338515 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:25.338518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:25.338653 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:25.339238 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:25.339244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:25.339463 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 19:12:25.339655 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 19:12:25.340152 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1907,"thread_start_us":1488,"threads_started":1}
I20260924 19:12:25.340869 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:25.323988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:39645 (local address 127.11.18.126:46147)
0924 19:12:25.324186 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:25.324191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:25.326273 (+ 2082us) server_negotiation.cc:407] Connection header received
0924 19:12:25.326541 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:25.326549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:25.326633 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:25.326746 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:25.327781 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.328556 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.330390 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.330861 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.331021 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:25.333315 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:25.333378 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:25.333385 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:25.333439 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:25.336968 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:25.338150 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:25.338158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:25.338164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:25.338341 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:25.338783 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:25.338789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:25.338793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:25.339104 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:25.339271 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:25.340239 (+ 968us) server_negotiation.cc:299] Negotiation successful
0924 19:12:25.340502 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:12:25.341701 13167 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:44897
I20260924 19:12:25.341737 13302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:44897 every 8 connection(s)
I20260924 19:12:25.343216 13167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:25.345847 13303 heartbeater.cc:347] Connected to a master server at 127.11.18.126:46147
I20260924 19:12:25.346100 13303 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:25.346819 13303 heartbeater.cc:511] Master 127.11.18.126:46147 requested a full tablet report, sending...
I20260924 19:12:25.348824 13095 ts_manager.cc:194] Registered new tserver with Master: 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897)
I20260924 19:12:25.350497 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:51035
I20260924 19:12:25.352808 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 13167
I20260924 19:12:25.353029 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:25.353936 11337 external_mini_cluster.cc:1428] Setting key d8836e4d2862410924dbfd778abedd10
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:12:25.397812 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46147
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:25.756414 13307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:25.756718 13307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:25.756881 13307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:25.765025 13307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:25.765166 13307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:25.765347 13307 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:12:25.765422 13307 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:12:25.765566 13307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:25.777148 13307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:46147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:25.778555 13307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:25.779958 13307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:25.795647 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:25.795483 13315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:25.795604 13313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:25.796646 13307 server_base.cc:1034] running on GCE node
I20260924 19:12:25.797430 13307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:25.798509 13307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:25.799729 13307 hybrid_clock.cc:648] HybridClock initialized: now 1790277145799662 us; error 60 us; skew 500 ppm
I20260924 19:12:25.800123 13307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:25 dist-test-slave-0lgr krb5kdc[13050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277145, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:25.803678 13307 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:12:25.805405 13307 webserver.cc:460] Webserver started at http://127.11.18.66:40905/ using document root <none> and password file <none>
I20260924 19:12:25.806166 13307 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:25.806275 13307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:25.806643 13307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:25.809711 13307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "0f1ffbee5cd348a2bce5680e18d129e6"
format_stamp: "Formatted at 2026-09-24 19:12:25 on dist-test-slave-0lgr"
server_key: "81a105c63c44c9ef5ecc44c86a1586b8"
server_key_iv: "61db4fcd45766e13b4bc1fca44a8e387"
server_key_version: "encryptionkey@0"
I20260924 19:12:25.810489 13307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "0f1ffbee5cd348a2bce5680e18d129e6"
format_stamp: "Formatted at 2026-09-24 19:12:25 on dist-test-slave-0lgr"
server_key: "81a105c63c44c9ef5ecc44c86a1586b8"
server_key_iv: "61db4fcd45766e13b4bc1fca44a8e387"
server_key_version: "encryptionkey@0"
I20260924 19:12:25.816035 13307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 19:12:25.820739 13322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:25.822639 13307 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 19:12:25.822871 13307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "0f1ffbee5cd348a2bce5680e18d129e6"
format_stamp: "Formatted at 2026-09-24 19:12:25 on dist-test-slave-0lgr"
server_key: "81a105c63c44c9ef5ecc44c86a1586b8"
server_key_iv: "61db4fcd45766e13b4bc1fca44a8e387"
server_key_version: "encryptionkey@0"
I20260924 19:12:25.823135 13307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:25.848528 13307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:25.853118 13307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:25.853472 13307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:25.856194 13307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:25.856333 13307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:25.856465 13307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:25.856573 13307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:12:25 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277145, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:25.880887 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:25.857059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:55225 (local address 127.11.18.126:46147)
0924 19:12:25.857662 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:25.857668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:25.858904 (+ 1236us) server_negotiation.cc:407] Connection header received
0924 19:12:25.861345 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:25.861351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:25.861443 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:25.861585 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:25.864058 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.865199 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.868235 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.868898 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.869044 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:25.874141 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:25.874149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:25.874159 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:25.874196 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:25.877302 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:25.878460 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:25.878466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:25.878470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:25.878609 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:25.879076 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:25.879081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:25.879084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:25.879358 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:25.879508 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:25.880271 (+ 763us) server_negotiation.cc:299] Negotiation successful
0924 19:12:25.880496 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":378,"threads_started":1}
I20260924 19:12:25.882299 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:25.857797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46147 (local address 127.11.18.66:55225)
0924 19:12:25.858701 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:25.858757 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:25.860982 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:25.861841 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:25.861853 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:25.862788 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:25.863761 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:25.863793 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.865523 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.865532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:25.868048 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:25.868061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.869176 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.869184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:25.869519 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:25.870446 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:25.870489 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:25.873855 (+ 3366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:25.877592 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:25.877600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:25.877616 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:25.878316 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:25.878734 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:25.878740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:25.878749 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:25.878951 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:25.879498 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:25.879506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:25.879895 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 19:12:25.880177 (+ 282us) client_negotiation.cc:240] Negotiation successful
0924 19:12:25.880724 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":370,"threads_started":1}
I20260924 19:12:25.912634 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:25.892728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:50319 (local address 127.11.18.126:46147)
0924 19:12:25.892964 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:25.892973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:25.894125 (+ 1152us) server_negotiation.cc:407] Connection header received
0924 19:12:25.894621 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:25.894627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:25.894786 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:25.894929 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:25.897086 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.898519 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.899925 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.900715 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.900867 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:25.903872 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:25.903879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:25.903883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:25.903920 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:25.907901 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:25.909657 (+ 1756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:25.909667 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:25.909674 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:25.909878 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:25.910526 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:25.910535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:25.910539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:25.910972 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:25.911185 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:25.911963 (+ 778us) server_negotiation.cc:299] Negotiation successful
0924 19:12:25.912388 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:12:25.912891 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:25.892163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46147 (local address 127.11.18.66:50319)
0924 19:12:25.893918 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:25.893943 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:25.894331 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:25.895435 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:25.895442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:25.896368 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:25.896868 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:25.896892 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.898706 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.898713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:25.899770 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:25.899783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:25.901018 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:25.901031 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:25.901297 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:25.902105 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:25.902144 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:25.903637 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:25.908261 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:25.908267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:25.908274 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:25.909395 (+ 1121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:25.910099 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:25.910107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:25.910113 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:25.910354 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:25.911120 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:25.911129 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:25.911581 (+ 452us) client_negotiation.cc:769] Sending connection context
0924 19:12:25.911801 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 19:12:25.912301 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1502,"thread_start_us":1251,"threads_started":1}
I20260924 19:12:25.914697 13307 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:42297
I20260924 19:12:25.914801 13442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:42297 every 8 connection(s)
I20260924 19:12:25.916400 13307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:25.918835 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 13307
I20260924 19:12:25.919029 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:25.919559 11337 external_mini_cluster.cc:1428] Setting key ab8b2fec166ee3c574e66ee2403fac92
I20260924 19:12:25.919616 13443 heartbeater.cc:347] Connected to a master server at 127.11.18.126:46147
I20260924 19:12:25.919900 13443 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:25.920652 13443 heartbeater.cc:511] Master 127.11.18.126:46147 requested a full tablet report, sending...
I20260924 19:12:25.922163 13095 ts_manager.cc:194] Registered new tserver with Master: 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297)
I20260924 19:12:25.923070 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:55225
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:12:25.966858 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46147
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:26.316817 13447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:26.317102 13447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:26.317204 13447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:26.324712 13447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:26.324837 13447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:26.324925 13447 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:12:26.324981 13447 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:12:26.325052 13447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:26.335364 13447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:46147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:26.336731 13447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:26.338222 13447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:26.353744 13452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:26.353782 13453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:26.353705 13455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:26.354368 13303 heartbeater.cc:503] Master 127.11.18.126:46147 was elected leader, sending a full tablet report...
I20260924 19:12:26.354727 13447 server_base.cc:1034] running on GCE node
I20260924 19:12:26.355525 13447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:26.356664 13447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:26.357972 13447 hybrid_clock.cc:648] HybridClock initialized: now 1790277146357923 us; error 37 us; skew 500 ppm
I20260924 19:12:26.358361 13447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:26 dist-test-slave-0lgr krb5kdc[13050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277146, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:26.366964 13447 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:12:26.368670 13447 webserver.cc:460] Webserver started at http://127.11.18.67:39361/ using document root <none> and password file <none>
I20260924 19:12:26.369518 13447 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:26.369637 13447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:26.369987 13447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:26.373311 13447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f"
format_stamp: "Formatted at 2026-09-24 19:12:26 on dist-test-slave-0lgr"
server_key: "c25e04c67c2160554f827dcf09c8f1a5"
server_key_iv: "cd183f9c93b025292cc7fe55cf255627"
server_key_version: "encryptionkey@0"
I20260924 19:12:26.374280 13447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f"
format_stamp: "Formatted at 2026-09-24 19:12:26 on dist-test-slave-0lgr"
server_key: "c25e04c67c2160554f827dcf09c8f1a5"
server_key_iv: "cd183f9c93b025292cc7fe55cf255627"
server_key_version: "encryptionkey@0"
I20260924 19:12:26.380309 13447 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 19:12:26.384733 13462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:26.386773 13447 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 19:12:26.386956 13447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f"
format_stamp: "Formatted at 2026-09-24 19:12:26 on dist-test-slave-0lgr"
server_key: "c25e04c67c2160554f827dcf09c8f1a5"
server_key_iv: "cd183f9c93b025292cc7fe55cf255627"
server_key_version: "encryptionkey@0"
I20260924 19:12:26.387226 13447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:26.403373 13447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:26.408435 13447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:26.408778 13447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:26.411929 13447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:26.412113 13447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:26.412276 13447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:26.412380 13447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:12:26 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277146, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:26.436995 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.412426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:51401 (local address 127.11.18.126:46147)
0924 19:12:26.412615 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.412621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.414290 (+ 1669us) server_negotiation.cc:407] Connection header received
0924 19:12:26.416433 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.416439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.416569 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.416685 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:26.419539 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.420686 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.423461 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.423973 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.424125 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.429218 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:26.429224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:26.429227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:26.429265 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:26.432380 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:26.433372 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:26.433379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:26.433384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:26.433575 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:26.434339 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:26.434347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:26.434350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:26.434804 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:26.434981 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.436115 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.436439 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:12:26.438133 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.413057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46147 (local address 127.11.18.67:51401)
0924 19:12:26.414081 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:26.414136 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:26.416123 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:26.416951 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:26.416960 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:26.418196 (+ 1236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:26.419337 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.419367 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.420862 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.420871 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.423187 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.423204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.424225 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.424232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.424488 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.425548 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:26.425574 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:26.428938 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:26.432587 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:26.432598 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:26.432617 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:26.433190 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:26.433730 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:26.433740 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:26.433752 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:26.434164 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:26.434951 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:26.434960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:26.435526 (+ 566us) client_negotiation.cc:769] Sending connection context
0924 19:12:26.435925 (+ 399us) client_negotiation.cc:240] Negotiation successful
0924 19:12:26.436650 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":506,"threads_started":1}
I20260924 19:12:26.460381 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.444141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46147 (local address 127.11.18.67:43063)
0924 19:12:26.444827 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:26.444843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:26.444958 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:26.445436 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:26.445443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:26.446103 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:26.446619 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.446631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.447974 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.447986 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.449599 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.449622 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.451113 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.451117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.451258 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.451974 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:26.452020 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:26.453179 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:26.456937 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:26.456942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:26.456947 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:26.457619 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:26.458109 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:26.458121 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:26.458129 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:26.458373 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:26.459209 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:26.459212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:26.459382 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 19:12:26.459516 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 19:12:26.459904 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":371,"threads_started":1}
I20260924 19:12:26.460386 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.444133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:43063 (local address 127.11.18.126:46147)
0924 19:12:26.444395 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.444405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.444927 (+ 522us) server_negotiation.cc:407] Connection header received
0924 19:12:26.445088 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.445097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.445210 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.445387 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:26.446776 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.447820 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.449794 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.450711 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.450908 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.453400 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:26.453409 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:26.453416 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:26.453475 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:26.456728 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:26.457741 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:26.457749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:26.457754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:26.457963 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:26.458621 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:26.458628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:26.458635 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:26.459078 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:26.459246 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.459681 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.459960 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:12:26.468529 13447 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:40277
I20260924 19:12:26.468562 13581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:40277 every 8 connection(s)
I20260924 19:12:26.469964 13447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:26.472298 13582 heartbeater.cc:347] Connected to a master server at 127.11.18.126:46147
I20260924 19:12:26.472507 13582 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:26.473142 13582 heartbeater.cc:511] Master 127.11.18.126:46147 requested a full tablet report, sending...
I20260924 19:12:26.474596 13095 ts_manager.cc:194] Registered new tserver with Master: e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277)
I20260924 19:12:26.475505 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:51401
I20260924 19:12:26.479972 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 13447
I20260924 19:12:26.480152 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:26.480638 11337 external_mini_cluster.cc:1428] Setting key e8742eec560b4a7f65a857e523e2db8f
I20260924 19:12:26.483947 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:12:26 dist-test-slave-0lgr krb5kdc[13050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:12:26 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:26.536202 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.520946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.11.18.126:46147)
0924 19:12:26.521165 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.521171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.521479 (+ 308us) server_negotiation.cc:407] Connection header received
0924 19:12:26.521720 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.521724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.521788 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.521893 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:26.523305 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.524067 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.525298 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.525747 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.525871 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.530401 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:26.530407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:26.530412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:26.530451 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:26.533320 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:26.534181 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:26.534186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:26.534188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:26.534319 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:26.534736 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:26.534740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:26.534742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:26.535079 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:26.535266 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.535644 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.535843 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:12:26.541630 13095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:12:26.548415 13095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:26.573239 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.564155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.11.18.65:44897)
0924 19:12:26.565891 (+ 1736us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.565898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.565925 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:12:26.566025 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.566033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.566256 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.566497 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:26.567101 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.568541 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.571070 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.572031 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.572195 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.572433 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.572709 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.572881 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1590,"thread_start_us":2077,"threads_started":1}
I20260924 19:12:26.573396 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.563863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:44897 (local address 127.0.0.1:41880)
0924 19:12:26.564574 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:26.564597 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:26.564748 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:26.566486 (+ 1738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:26.566491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:26.566515 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:26.566919 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.566930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.568708 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.568713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.570922 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.570932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.572250 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.572256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.572475 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.572526 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 19:12:26.572667 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 19:12:26.572881 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":86,"thread_start_us":391,"threads_started":1}
I20260924 19:12:26.573921 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.562367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:42297 (local address 127.0.0.1:46644)
0924 19:12:26.563487 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:26.563523 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:26.563694 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:26.564995 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:26.565005 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:26.565043 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:26.565559 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.565576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.567875 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.567881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.570598 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.570612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.572391 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.572400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.572584 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.572623 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:12:26.572685 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:12:26.572882 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":101,"thread_start_us":720,"threads_started":1}
I20260924 19:12:26.574158 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.562774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.11.18.66:42297)
0924 19:12:26.564303 (+ 1529us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.564312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.564347 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:12:26.564444 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.564454 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.564745 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.565835 (+ 1090us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:26.566100 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.567722 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.570767 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.572142 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.573410 (+ 1268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.573610 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.573680 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.573851 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1332,"thread_start_us":1170,"threads_started":1}
I20260924 19:12:26.576752 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.564576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:40277 (local address 127.0.0.1:52050)
0924 19:12:26.565257 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:26.565286 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:26.565441 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:26.567631 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:26.567636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:26.567675 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:26.568189 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.568209 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.570271 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.570279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.572330 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.572343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.575925 (+ 3582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.575933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.576108 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.576141 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:12:26.576242 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:12:26.576376 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":3,"thread_start_us":786,"threads_started":1}
I20260924 19:12:26.576787 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.565112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.11.18.67:40277)
0924 19:12:26.567185 (+ 2073us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.567191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.567215 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:12:26.567284 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.567289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.567455 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.567642 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:26.568429 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.570135 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.572525 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.575665 (+ 3140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.575809 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.576029 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.576295 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.576453 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1930,"thread_start_us":1718,"threads_started":1}
I20260924 19:12:26.577744 13373 tablet_service.cc:1467] Processing CreateTablet for tablet e432ed07e99a461b8f891f67e0a5fc37 (DEFAULT_TABLE table=test-table [id=273721279be04379a405727b61e320de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:26.577723 13233 tablet_service.cc:1467] Processing CreateTablet for tablet e432ed07e99a461b8f891f67e0a5fc37 (DEFAULT_TABLE table=test-table [id=273721279be04379a405727b61e320de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:26.579172 13233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e432ed07e99a461b8f891f67e0a5fc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:26.579200 13373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e432ed07e99a461b8f891f67e0a5fc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:26.580657 13512 tablet_service.cc:1467] Processing CreateTablet for tablet e432ed07e99a461b8f891f67e0a5fc37 (DEFAULT_TABLE table=test-table [id=273721279be04379a405727b61e320de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:26.582079 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e432ed07e99a461b8f891f67e0a5fc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:26.591599 13601 tablet_bootstrap.cc:492] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Bootstrap starting.
I20260924 19:12:26.595254 13602 tablet_bootstrap.cc:492] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Bootstrap starting.
I20260924 19:12:26.597817 13603 tablet_bootstrap.cc:492] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Bootstrap starting.
I20260924 19:12:26.597997 13601 tablet_bootstrap.cc:654] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:26.599218 13602 tablet_bootstrap.cc:654] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:26.599893 13601 log.cc:824] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:26.600593 13602 log.cc:824] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:26.600993 13603 tablet_bootstrap.cc:654] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:26.602213 13603 log.cc:824] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:26.603464 13601 tablet_bootstrap.cc:492] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: No bootstrap required, opened a new log
I20260924 19:12:26.603791 13601 ts_tablet_manager.cc:1397] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 19:12:26.605595 13603 tablet_bootstrap.cc:492] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: No bootstrap required, opened a new log
I20260924 19:12:26.605880 13603 ts_tablet_manager.cc:1397] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 19:12:26.606181 13602 tablet_bootstrap.cc:492] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: No bootstrap required, opened a new log
I20260924 19:12:26.606475 13602 ts_tablet_manager.cc:1397] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260924 19:12:26.613068 13603 raft_consensus.cc:348] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.613652 13603 raft_consensus.cc:374] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:26.613785 13603 raft_consensus.cc:729] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6326fc66bbe4eae9b10a29ee95f9a4f, State: Initialized, Role: FOLLOWER
I20260924 19:12:26.615065 13603 consensus_queue.cc:260] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.615664 13607 raft_consensus.cc:482] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:26.615989 13601 raft_consensus.cc:348] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.616075 13607 raft_consensus.cc:504] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.616593 13601 raft_consensus.cc:374] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:26.616739 13601 raft_consensus.cc:729] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1ffbee5cd348a2bce5680e18d129e6, State: Initialized, Role: FOLLOWER
I20260924 19:12:26.617565 13601 consensus_queue.cc:260] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.618196 13602 raft_consensus.cc:348] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.618896 13602 raft_consensus.cc:374] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:26.619088 13602 raft_consensus.cc:729] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, State: Initialized, Role: FOLLOWER
I20260924 19:12:26.619987 13602 consensus_queue.cc:260] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.622177 13443 heartbeater.cc:503] Master 127.11.18.126:46147 was elected leader, sending a full tablet report...
I20260924 19:12:26.623487 13601 ts_tablet_manager.cc:1428] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent starting tablet: real 0.020s user 0.017s sys 0.000s
I20260924 19:12:26.624537 13582 heartbeater.cc:503] Master 127.11.18.126:46147 was elected leader, sending a full tablet report...
I20260924 19:12:26.625155 13603 ts_tablet_manager.cc:1428] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260924 19:12:26.628360 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.619286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:42297 (local address 127.11.18.67:50829)
0924 19:12:26.619894 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:26.619923 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:26.620079 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:26.621464 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:26.621472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:26.621514 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:26.621809 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.621825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.623949 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.623955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.625828 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.625842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.627508 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.627516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.627744 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.627779 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:12:26.627867 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:12:26.628032 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410}
I20260924 19:12:26.629580 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.620460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:50829 (local address 127.11.18.66:42297)
0924 19:12:26.620782 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.620791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.620817 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:12:26.620908 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.620916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.621033 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.621144 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:26.622023 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.623701 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.626009 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.627178 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.627345 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.627419 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.628965 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.629181 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:12:26.633137 13607 leader_election.cc:288] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297), 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897)
I20260924 19:12:26.635078 13602 ts_tablet_manager.cc:1428] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent starting tablet: real 0.028s user 0.013s sys 0.006s
I20260924 19:12:26.637059 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e432ed07e99a461b8f891f67e0a5fc37" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" is_pre_election: true
I20260924 19:12:26.637691 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.627819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:44897 (local address 127.11.18.67:51529)
0924 19:12:26.628628 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:26.628654 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:26.628791 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:26.629970 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:26.629978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:26.630017 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:26.630295 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.630310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.632235 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.632243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.634791 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.634812 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.636649 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.636655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.636866 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.636929 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 19:12:26.637045 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:12:26.637221 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":7918,"threads_started":1}
I20260924 19:12:26.637701 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.628397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:51529 (local address 127.11.18.65:44897)
0924 19:12:26.628649 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.628656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.629627 (+ 971us) server_negotiation.cc:407] Connection header received
0924 19:12:26.629724 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.629730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.629806 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.629935 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:26.630512 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.631991 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.635074 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.636402 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.636592 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.636675 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.637169 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.637401 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:12:26.637881 13398 raft_consensus.cc:2454] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 0.
I20260924 19:12:26.638759 13466 leader_election.cc:302] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1ffbee5cd348a2bce5680e18d129e6, e6326fc66bbe4eae9b10a29ee95f9a4f; no voters:
I20260924 19:12:26.639115 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e432ed07e99a461b8f891f67e0a5fc37" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" is_pre_election: true
I20260924 19:12:26.639353 13607 raft_consensus.cc:2781] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:26.639631 13607 raft_consensus.cc:482] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:26.639787 13607 raft_consensus.cc:3037] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:26.639735 13253 raft_consensus.cc:2454] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 0.
I20260924 19:12:26.642815 13607 raft_consensus.cc:504] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.643592 13607 leader_election.cc:288] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 election: Requested vote from peers 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297), 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897)
I20260924 19:12:26.644155 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e432ed07e99a461b8f891f67e0a5fc37" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6"
I20260924 19:12:26.644227 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e432ed07e99a461b8f891f67e0a5fc37" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560"
I20260924 19:12:26.644411 13398 raft_consensus.cc:3037] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:26.644462 13253 raft_consensus.cc:3037] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:26.646924 13398 raft_consensus.cc:2454] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 1.
I20260924 19:12:26.646938 13253 raft_consensus.cc:2454] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 1.
I20260924 19:12:26.647447 13466 leader_election.cc:302] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1ffbee5cd348a2bce5680e18d129e6, e6326fc66bbe4eae9b10a29ee95f9a4f; no voters:
I20260924 19:12:26.647823 13607 raft_consensus.cc:2781] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:26.648080 13607 raft_consensus.cc:686] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 LEADER]: Becoming Leader. State: Replica: e6326fc66bbe4eae9b10a29ee95f9a4f, State: Running, Role: LEADER
I20260924 19:12:26.648564 13607 consensus_queue.cc:237] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.653730 13095 catalog_manager.cc:5495] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f reported cstate change: term changed from 0 to 1, leader changed from <none> to e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67). New cstate: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } health_report { overall_health: HEALTHY } } }
W20260924 19:12:26.670579 13444 tablet.cc:2369] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:12:26 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277144, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
W20260924 19:12:26.721674 13583 tablet.cc:2369] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:12:26.727734 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.708570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46147 (local address 127.0.0.1:47404)
0924 19:12:26.709404 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:26.709425 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:26.709563 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:26.709989 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:26.709994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:26.710918 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:26.711319 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.711331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.712970 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.712976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.714619 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:26.714633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.715820 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.715826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:26.716037 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.716912 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:26.716934 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:26.719974 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:26.724534 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:26.724547 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:26.724556 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:26.725165 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:26.725672 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:26.725677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:26.725681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:26.725826 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:26.726567 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:26.726573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:26.726721 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 19:12:26.726994 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 19:12:26.727429 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":166,"thread_start_us":463,"threads_started":1}
I20260924 19:12:26.727816 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:26.708504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.11.18.126:46147)
0924 19:12:26.708727 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:26.708734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:26.709527 (+ 793us) server_negotiation.cc:407] Connection header received
0924 19:12:26.709696 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:26.709703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:26.709818 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:26.709941 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:26.711494 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.712817 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.714787 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:26.715621 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:26.715765 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:26.720201 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:26.720210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:26.720216 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:26.720260 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:26.724369 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:26.725316 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:26.725348 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:26.725352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:26.725549 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:26.725947 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:26.725954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:26.725958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:26.726363 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:26.726626 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:26.727032 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 19:12:26.727305 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:12:26.733189 13095 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:47404:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:12:26.734300 13095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:26.742064 13233 tablet_service.cc:1467] Processing CreateTablet for tablet 12d10ba18b15493990c697251598008f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c62f300cbf64324a09828dbf97d2644]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:12:26.742293 13373 tablet_service.cc:1467] Processing CreateTablet for tablet 12d10ba18b15493990c697251598008f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c62f300cbf64324a09828dbf97d2644]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:12:26.742570 13512 tablet_service.cc:1467] Processing CreateTablet for tablet 12d10ba18b15493990c697251598008f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c62f300cbf64324a09828dbf97d2644]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:12:26.743161 13233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d10ba18b15493990c697251598008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:26.743269 13373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d10ba18b15493990c697251598008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:26.743314 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d10ba18b15493990c697251598008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:26.750185 13601 tablet_bootstrap.cc:492] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: Bootstrap starting.
I20260924 19:12:26.750309 13602 tablet_bootstrap.cc:492] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Bootstrap starting.
I20260924 19:12:26.750708 13603 tablet_bootstrap.cc:492] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: Bootstrap starting.
I20260924 19:12:26.752028 13601 tablet_bootstrap.cc:654] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:26.752038 13602 tablet_bootstrap.cc:654] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:26.752430 13603 tablet_bootstrap.cc:654] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:26.755613 13603 tablet_bootstrap.cc:492] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: No bootstrap required, opened a new log
I20260924 19:12:26.755808 13603 ts_tablet_manager.cc:1397] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:12:26.756814 13603 raft_consensus.cc:348] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.757129 13603 raft_consensus.cc:374] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:26.757242 13603 raft_consensus.cc:729] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6326fc66bbe4eae9b10a29ee95f9a4f, State: Initialized, Role: FOLLOWER
I20260924 19:12:26.758020 13603 consensus_queue.cc:260] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.758873 13603 ts_tablet_manager.cc:1428] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260924 19:12:26.758693 13607 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:26.759119 13607 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:26.763191 13602 tablet_bootstrap.cc:492] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: No bootstrap required, opened a new log
I20260924 19:12:26.763389 13602 ts_tablet_manager.cc:1397] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260924 19:12:26.764015 13602 raft_consensus.cc:348] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.764196 13602 raft_consensus.cc:374] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:26.764254 13602 raft_consensus.cc:729] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, State: Initialized, Role: FOLLOWER
I20260924 19:12:26.764355 13601 tablet_bootstrap.cc:492] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: No bootstrap required, opened a new log
I20260924 19:12:26.764465 13602 consensus_queue.cc:260] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.764549 13601 ts_tablet_manager.cc:1397] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260924 19:12:26.764894 13610 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:26.765120 13602 ts_tablet_manager.cc:1428] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 19:12:26.765359 13610 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:26.766230 13601 raft_consensus.cc:348] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.766489 13601 raft_consensus.cc:374] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:26.766598 13601 raft_consensus.cc:729] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1ffbee5cd348a2bce5680e18d129e6, State: Initialized, Role: FOLLOWER
I20260924 19:12:26.766850 13601 consensus_queue.cc:260] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:26.767356 13609 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:26.767741 13609 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:26.768213 13601 ts_tablet_manager.cc:1428] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
W20260924 19:12:26.846554 13304 tablet.cc:2369] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:12:27.120437 13607 consensus_queue.cc:1035] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:27.129106 13624 consensus_queue.cc:1035] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:27.196130 13624 raft_consensus.cc:482] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:27.196382 13624 raft_consensus.cc:504] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.197026 13624 leader_election.cc:288] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897), 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297)
I20260924 19:12:27.197733 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d10ba18b15493990c697251598008f" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" is_pre_election: true
I20260924 19:12:27.197760 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d10ba18b15493990c697251598008f" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" is_pre_election: true
I20260924 19:12:27.197978 13253 raft_consensus.cc:2454] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 0.
I20260924 19:12:27.198109 13398 raft_consensus.cc:2454] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 0.
I20260924 19:12:27.198544 13466 leader_election.cc:302] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1ffbee5cd348a2bce5680e18d129e6, e6326fc66bbe4eae9b10a29ee95f9a4f; no voters:
I20260924 19:12:27.198836 13624 raft_consensus.cc:2781] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:27.199028 13624 raft_consensus.cc:482] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:27.199164 13624 raft_consensus.cc:3037] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:27.201146 13624 raft_consensus.cc:504] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.201671 13624 leader_election.cc:288] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 election: Requested vote from peers 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897), 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297)
I20260924 19:12:27.202265 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d10ba18b15493990c697251598008f" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560"
I20260924 19:12:27.202342 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d10ba18b15493990c697251598008f" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6"
I20260924 19:12:27.202561 13253 raft_consensus.cc:3037] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:27.202656 13398 raft_consensus.cc:3037] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:27.204526 13253 raft_consensus.cc:2454] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 1.
I20260924 19:12:27.204946 13466 leader_election.cc:302] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, e6326fc66bbe4eae9b10a29ee95f9a4f; no voters:
I20260924 19:12:27.204979 13398 raft_consensus.cc:2454] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 1.
I20260924 19:12:27.205238 13624 raft_consensus.cc:2781] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:27.205430 13624 raft_consensus.cc:686] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 LEADER]: Becoming Leader. State: Replica: e6326fc66bbe4eae9b10a29ee95f9a4f, State: Running, Role: LEADER
I20260924 19:12:27.205709 13624 consensus_queue.cc:237] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.207249 13608 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: TxnStatusTablet state changed. Reason: New leader e6326fc66bbe4eae9b10a29ee95f9a4f. Latest consensus state: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:27.207518 13608 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:12:27.208114 13629 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:12:27.208439 13629 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:12:27.208912 13095 catalog_manager.cc:5495] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f reported cstate change: term changed from 0 to 1, leader changed from <none> to e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67). New cstate: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } health_report { overall_health: HEALTHY } } }
I20260924 19:12:27.345641 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.327601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:40277 (local address 127.0.0.1:52056)
0924 19:12:27.328409 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:27.328432 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:27.328615 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:27.338530 (+ 9915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:27.338535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:27.338550 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:12:27.338772 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.338781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.340675 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.340683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.342336 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.342358 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.343466 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.343473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.343647 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.343738 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.344974 (+ 1236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.345003 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 19:12:27.345112 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 19:12:27.345268 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":415,"threads_started":1}
I20260924 19:12:27.345659 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.327695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.11.18.67:40277)
0924 19:12:27.337915 (+ 10220us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:27.337925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:27.337950 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:12:27.338074 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:27.338081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:27.338214 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:27.338437 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:27.338966 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.340471 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.342526 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.343274 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.343440 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.343957 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.344835 (+ 878us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.345006 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:27.345157 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 19:12:27.345260 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10041,"thread_start_us":9844,"threads_started":1}
I20260924 19:12:27.354712 13253 raft_consensus.cc:1261] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Refusing update from remote peer e6326fc66bbe4eae9b10a29ee95f9a4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:27.354718 13398 raft_consensus.cc:1261] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Refusing update from remote peer e6326fc66bbe4eae9b10a29ee95f9a4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:27.355473 13608 consensus_queue.cc:1035] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:27.355852 13624 consensus_queue.cc:1035] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:27.363354 13609 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: TxnStatusTablet state changed. Reason: New leader e6326fc66bbe4eae9b10a29ee95f9a4f. Latest consensus state: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:27.363723 13609 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:27.367060 13623 mvcc.cc:205] Tried to move back new op lower bound from 7332975195555680256 to 7332975194956509184. Current Snapshot: MvccSnapshot[applied={T|T < 7332975195555680256}]
I20260924 19:12:27.369431 13609 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:27.369711 13609 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P 0f1ffbee5cd348a2bce5680e18d129e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:27.370704 13610 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: TxnStatusTablet state changed. Reason: New leader e6326fc66bbe4eae9b10a29ee95f9a4f. Latest consensus state: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:27.370968 13610 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:27.373946 13610 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:27.374200 13610 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:12:27.378173 13607 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:27.378471 13607 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:12:27.378132 13624 tablet_replica.cc:439] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } } }
I20260924 19:12:27.378871 13624 tablet_replica.cc:442] T 12d10ba18b15493990c697251598008f P e6326fc66bbe4eae9b10a29ee95f9a4f: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:12:27.391451 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.385955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.11.18.67:40277)
0924 19:12:27.386223 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:27.386232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:27.386579 (+ 347us) server_negotiation.cc:407] Connection header received
0924 19:12:27.386810 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:27.386814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:27.386882 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:27.386991 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:27.387409 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.388197 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.389632 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.390047 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.390187 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.390658 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.390760 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.390861 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:27.391020 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 19:12:27.391102 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 19:12:27.410511 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.403543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:36889 (local address 127.11.18.67:40277)
0924 19:12:27.403751 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:27.403759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:27.404569 (+ 810us) server_negotiation.cc:407] Connection header received
0924 19:12:27.404874 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:27.404878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:27.404950 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:27.405066 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:27.405730 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.406598 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.408310 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.408794 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.408939 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.409535 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.409669 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.409787 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:27.409948 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:12:27.410032 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:12:27.410614 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.403567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:40277 (local address 127.11.18.67:36889)
0924 19:12:27.404456 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:27.404474 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:27.404667 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:27.405077 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:27.405082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:27.405093 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:12:27.405541 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.405553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.406758 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.406764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.408160 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.408174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.409021 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.409035 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.409157 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.409217 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.409780 (+ 563us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.409790 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:12:27.409914 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 19:12:27.410071 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":469,"threads_started":1}
I20260924 19:12:27.499012 13095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:27.499337 13095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:27.503887 13095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: Sending DeleteTablet for 3 replicas of tablet e432ed07e99a461b8f891f67e0a5fc37
I20260924 19:12:27.505225 13373 tablet_service.cc:1514] Processing DeleteTablet for tablet e432ed07e99a461b8f891f67e0a5fc37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:27 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:46644
I20260924 19:12:27.505416 13233 tablet_service.cc:1514] Processing DeleteTablet for tablet e432ed07e99a461b8f891f67e0a5fc37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:27 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:41880
I20260924 19:12:27.506276 13646 tablet_replica.cc:330] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: stopping tablet replica
I20260924 19:12:27.506425 13647 tablet_replica.cc:330] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: stopping tablet replica
I20260924 19:12:27.506851 13647 raft_consensus.cc:2229] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:27.506851 13646 raft_consensus.cc:2229] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:27.507261 13646 raft_consensus.cc:2258] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:27.507261 13647 raft_consensus.cc:2258] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:27.507923 13095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:12:27.508661 13095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:27.510253 13646 ts_tablet_manager.cc:1905] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:27.510253 13647 ts_tablet_manager.cc:1905] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:27.510236 13512 tablet_service.cc:1514] Processing DeleteTablet for tablet e432ed07e99a461b8f891f67e0a5fc37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:27 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:52050
I20260924 19:12:27.517944 13646 ts_tablet_manager.cc:1918] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:27.518309 13511 tablet_service.cc:1467] Processing CreateTablet for tablet 8981d15378cc4c4e8054dfe3dc4230cd (DEFAULT_TABLE table=test-table [id=0b95087e93b1405699ebd050734aeb64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:27.518821 13648 tablet_replica.cc:330] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: stopping tablet replica
I20260924 19:12:27.518951 13511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8981d15378cc4c4e8054dfe3dc4230cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:27.518965 13647 ts_tablet_manager.cc:1918] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:27.519122 13647 log.cc:1196] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/e432ed07e99a461b8f891f67e0a5fc37
I20260924 19:12:27.519585 13648 raft_consensus.cc:2229] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:12:27.519760 13647 ts_tablet_manager.cc:1939] T e432ed07e99a461b8f891f67e0a5fc37 P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting consensus metadata
I20260924 19:12:27.520470 13648 raft_consensus.cc:2258] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:27.518137 13373 tablet_service.cc:1467] Processing CreateTablet for tablet 8981d15378cc4c4e8054dfe3dc4230cd (DEFAULT_TABLE table=test-table [id=0b95087e93b1405699ebd050734aeb64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:27.520795 13646 log.cc:1196] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/e432ed07e99a461b8f891f67e0a5fc37
I20260924 19:12:27.520993 13373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8981d15378cc4c4e8054dfe3dc4230cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:27.521498 13646 ts_tablet_manager.cc:1939] T e432ed07e99a461b8f891f67e0a5fc37 P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting consensus metadata
I20260924 19:12:27.522113 13233 tablet_service.cc:1467] Processing CreateTablet for tablet 8981d15378cc4c4e8054dfe3dc4230cd (DEFAULT_TABLE table=test-table [id=0b95087e93b1405699ebd050734aeb64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:27.522562 13233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8981d15378cc4c4e8054dfe3dc4230cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:27.524097 13648 ts_tablet_manager.cc:1905] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:27.525731 13082 catalog_manager.cc:4841] TS 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297): tablet e432ed07e99a461b8f891f67e0a5fc37 (table test-table [id=273721279be04379a405727b61e320de]) successfully deleted
I20260924 19:12:27.528528 13082 catalog_manager.cc:4841] TS 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897): tablet e432ed07e99a461b8f891f67e0a5fc37 (table test-table [id=273721279be04379a405727b61e320de]) successfully deleted
I20260924 19:12:27.534483 13649 tablet_bootstrap.cc:492] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: Bootstrap starting.
I20260924 19:12:27.536182 13648 ts_tablet_manager.cc:1918] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:27.536367 13649 tablet_bootstrap.cc:654] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:27.536542 13648 log.cc:1196] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/e432ed07e99a461b8f891f67e0a5fc37
I20260924 19:12:27.537226 13648 ts_tablet_manager.cc:1939] T e432ed07e99a461b8f891f67e0a5fc37 P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting consensus metadata
I20260924 19:12:27.539189 13082 catalog_manager.cc:4841] TS e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277): tablet e432ed07e99a461b8f891f67e0a5fc37 (table test-table [id=273721279be04379a405727b61e320de]) successfully deleted
I20260924 19:12:27.539290 13649 tablet_bootstrap.cc:492] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: No bootstrap required, opened a new log
I20260924 19:12:27.539431 13649 ts_tablet_manager.cc:1397] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 19:12:27.540160 13649 raft_consensus.cc:348] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.540388 13649 raft_consensus.cc:374] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:27.540475 13649 raft_consensus.cc:729] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1ffbee5cd348a2bce5680e18d129e6, State: Initialized, Role: FOLLOWER
I20260924 19:12:27.540776 13649 consensus_queue.cc:260] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.541441 13649 ts_tablet_manager.cc:1428] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:12:27.542135 13651 tablet_bootstrap.cc:492] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Bootstrap starting.
I20260924 19:12:27.543272 13650 tablet_bootstrap.cc:492] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: Bootstrap starting.
I20260924 19:12:27.545487 13650 tablet_bootstrap.cc:654] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:27.546224 13651 tablet_bootstrap.cc:654] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:27.547816 13650 tablet_bootstrap.cc:492] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: No bootstrap required, opened a new log
I20260924 19:12:27.547986 13650 ts_tablet_manager.cc:1397] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 19:12:27.548923 13650 raft_consensus.cc:348] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.549186 13650 raft_consensus.cc:374] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:27.549292 13650 raft_consensus.cc:729] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6326fc66bbe4eae9b10a29ee95f9a4f, State: Initialized, Role: FOLLOWER
I20260924 19:12:27.549604 13650 consensus_queue.cc:260] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.550259 13650 ts_tablet_manager.cc:1428] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:12:27.551146 13651 tablet_bootstrap.cc:492] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: No bootstrap required, opened a new log
I20260924 19:12:27.551307 13651 ts_tablet_manager.cc:1397] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260924 19:12:27.552114 13651 raft_consensus.cc:348] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.552358 13651 raft_consensus.cc:374] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:27.552640 13651 raft_consensus.cc:729] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, State: Initialized, Role: FOLLOWER
I20260924 19:12:27.552934 13651 consensus_queue.cc:260] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.554333 13651 ts_tablet_manager.cc:1428] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:12:27.684986 13610 raft_consensus.cc:482] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:27.685269 13610 raft_consensus.cc:504] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.687060 13610 leader_election.cc:288] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297), e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277)
I20260924 19:12:27.696825 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.687543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:43433 (local address 127.11.18.66:42297)
0924 19:12:27.689667 (+ 2124us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:27.689673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:27.689696 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:12:27.689762 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:27.689767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:27.689846 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:27.689958 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:27.690892 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.692189 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.694215 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.695582 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.695824 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.695921 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:27.696246 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 19:12:27.696417 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1983,"thread_start_us":1790,"threads_started":1}
I20260924 19:12:27.696871 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.687317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:42297 (local address 127.11.18.65:43433)
0924 19:12:27.688089 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:27.688112 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:27.688252 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:27.690015 (+ 1763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:27.690021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:27.690056 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:27.690587 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.690601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.692328 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.692337 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.694055 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.694068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.695827 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.695835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.696040 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.696082 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:12:27.696197 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 19:12:27.696360 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":30,"thread_start_us":317,"threads_started":1}
I20260924 19:12:27.697902 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.688687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:37761 (local address 127.11.18.67:40277)
0924 19:12:27.688978 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:27.688987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:27.689189 (+ 202us) server_negotiation.cc:407] Connection header received
0924 19:12:27.689284 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:27.689292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:27.689415 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:27.689528 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:27.690453 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.691732 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.694572 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.696131 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.696338 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.696421 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:27.697454 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0924 19:12:27.697603 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:12:27.697813 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8981d15378cc4c4e8054dfe3dc4230cd" candidate_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" is_pre_election: true
I20260924 19:12:27.698092 13398 raft_consensus.cc:2454] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bbcf3a78ebe4c1da84e9ef2eb7c4560 in term 0.
I20260924 19:12:27.698760 13186 leader_election.cc:302] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1ffbee5cd348a2bce5680e18d129e6, 9bbcf3a78ebe4c1da84e9ef2eb7c4560; no voters:
I20260924 19:12:27.699215 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.688214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:40277 (local address 127.11.18.65:37761)
0924 19:12:27.688958 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:27.688977 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:27.689105 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:27.689719 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:27.689726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:27.689751 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:27.690240 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.690256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.691912 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.691918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.694390 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.694407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.696583 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.696591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.696828 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.696880 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 19:12:27.698443 (+ 1563us) client_negotiation.cc:240] Negotiation successful
0924 19:12:27.698604 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":23,"thread_start_us":773,"threads_started":1}
I20260924 19:12:27.699324 13610 raft_consensus.cc:2781] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:27.699463 13610 raft_consensus.cc:482] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:27.699575 13610 raft_consensus.cc:3037] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:27.700170 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8981d15378cc4c4e8054dfe3dc4230cd" candidate_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" is_pre_election: true
I20260924 19:12:27.700664 13537 raft_consensus.cc:2454] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bbcf3a78ebe4c1da84e9ef2eb7c4560 in term 0.
I20260924 19:12:27.702245 13610 raft_consensus.cc:504] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.702859 13610 leader_election.cc:288] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [CANDIDATE]: Term 1 election: Requested vote from peers 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297), e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277)
I20260924 19:12:27.703322 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8981d15378cc4c4e8054dfe3dc4230cd" candidate_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6"
I20260924 19:12:27.703459 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8981d15378cc4c4e8054dfe3dc4230cd" candidate_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f"
I20260924 19:12:27.703585 13398 raft_consensus.cc:3037] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:27.703734 13537 raft_consensus.cc:3037] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:27.706316 13537 raft_consensus.cc:2454] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bbcf3a78ebe4c1da84e9ef2eb7c4560 in term 1.
I20260924 19:12:27.706782 13186 leader_election.cc:302] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, e6326fc66bbe4eae9b10a29ee95f9a4f; no voters:
I20260924 19:12:27.706991 13398 raft_consensus.cc:2454] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bbcf3a78ebe4c1da84e9ef2eb7c4560 in term 1.
I20260924 19:12:27.707140 13610 raft_consensus.cc:2781] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:27.709623 13610 raft_consensus.cc:686] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 LEADER]: Becoming Leader. State: Replica: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, State: Running, Role: LEADER
I20260924 19:12:27.710083 13610 consensus_queue.cc:237] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:27.714061 13095 catalog_manager.cc:5495] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65). New cstate: current_term: 1 leader_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } health_report { overall_health: UNKNOWN } } }
I20260924 19:12:27.756824 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.750703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.11.18.65:44897)
0924 19:12:27.751600 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:27.751605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:27.751632 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:12:27.751715 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:27.751718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:27.751788 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:27.751961 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:27.752377 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.753154 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.754526 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.754953 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.755105 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.755532 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.756171 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.756340 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:27.756399 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 19:12:27.756476 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":628,"threads_started":1}
I20260924 19:12:27.775151 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.767888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:40277 (local address 127.11.18.65:43075)
0924 19:12:27.768675 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:27.768695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:27.768818 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:27.769263 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:27.769268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:27.769278 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:12:27.769656 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.769665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.771271 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.771276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.772664 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.772677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.773882 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.773887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.774012 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.774071 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.774603 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.774623 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 19:12:27.774702 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:12:27.774816 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":398,"threads_started":1}
I20260924 19:12:27.775242 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.767960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:43075 (local address 127.11.18.67:40277)
0924 19:12:27.768182 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:27.768189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:27.768781 (+ 592us) server_negotiation.cc:407] Connection header received
0924 19:12:27.768931 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:27.768937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:27.769119 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:27.769238 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:27.769827 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.771092 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.772811 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.773665 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.773814 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.774250 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.774400 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.774506 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:27.774750 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 19:12:27.774846 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:12:27.785993 13537 raft_consensus.cc:1261] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Refusing update from remote peer 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:27.785897 13398 raft_consensus.cc:1261] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Refusing update from remote peer 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:27.786791 13658 consensus_queue.cc:1035] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:27.787371 13610 consensus_queue.cc:1035] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:27.795495 13625 mvcc.cc:205] Tried to move back new op lower bound from 7332975197324513280 to 7332975197024796672. Current Snapshot: MvccSnapshot[applied={T|T < 7332975197324513280}]
I20260924 19:12:27.796602 13627 mvcc.cc:205] Tried to move back new op lower bound from 7332975197324513280 to 7332975197024796672. Current Snapshot: MvccSnapshot[applied={T|T < 7332975197324513280}]
I20260924 19:12:27.822949 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.816152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:44897 (local address 127.11.18.67:57315)
0924 19:12:27.816349 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:27.816379 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:27.816527 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:27.817070 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:27.817075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:27.817090 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:12:27.817376 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.817390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.818968 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.818977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.820629 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:27.820643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.821761 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.821768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:27.821955 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.821996 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.822483 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.822496 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:12:27.822564 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:12:27.822686 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:12:27.823582 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:27.816560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:57315 (local address 127.11.18.65:44897)
0924 19:12:27.816761 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:27.816767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:27.816788 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:12:27.816854 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:27.816862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:27.816945 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:27.817047 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:27.817574 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.818752 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.820841 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:27.821612 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:27.821753 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:27.822167 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:27.822331 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:27.823222 (+ 891us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:27.823276 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:12:27.823348 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 19:12:27.824852 13625 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:12:29.437433 13095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:29.437722 13095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:29.442195 13095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: Sending DeleteTablet for 3 replicas of tablet 8981d15378cc4c4e8054dfe3dc4230cd
I20260924 19:12:29.443198 13233 tablet_service.cc:1514] Processing DeleteTablet for tablet 8981d15378cc4c4e8054dfe3dc4230cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:29 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:41880
I20260924 19:12:29.443385 13373 tablet_service.cc:1514] Processing DeleteTablet for tablet 8981d15378cc4c4e8054dfe3dc4230cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:29 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:46644
I20260924 19:12:29.443408 13511 tablet_service.cc:1514] Processing DeleteTablet for tablet 8981d15378cc4c4e8054dfe3dc4230cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:29 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:52050
I20260924 19:12:29.454612 13691 tablet_replica.cc:330] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: stopping tablet replica
I20260924 19:12:29.454861 13691 raft_consensus.cc:2229] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:12:29.455535 13691 raft_consensus.cc:2258] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:29.456146 13692 tablet_replica.cc:330] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: stopping tablet replica
I20260924 19:12:29.456350 13692 raft_consensus.cc:2229] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:29.456561 13692 raft_consensus.cc:2258] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:29.456640 13691 ts_tablet_manager.cc:1905] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:29.457567 13692 ts_tablet_manager.cc:1905] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:29.459750 13693 tablet_replica.cc:330] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: stopping tablet replica
I20260924 19:12:29.460026 13693 raft_consensus.cc:2229] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:29.460280 13693 raft_consensus.cc:2258] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:29.461311 13693 ts_tablet_manager.cc:1905] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:29.463630 13691 ts_tablet_manager.cc:1918] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:12:29.464211 13691 log.cc:1196] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/8981d15378cc4c4e8054dfe3dc4230cd
I20260924 19:12:29.464649 13691 ts_tablet_manager.cc:1939] T 8981d15378cc4c4e8054dfe3dc4230cd P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting consensus metadata
I20260924 19:12:29.465548 13692 ts_tablet_manager.cc:1918] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:12:29.465682 13692 log.cc:1196] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/8981d15378cc4c4e8054dfe3dc4230cd
I20260924 19:12:29.465853 13082 catalog_manager.cc:4841] TS 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897): tablet 8981d15378cc4c4e8054dfe3dc4230cd (table test-table [id=0b95087e93b1405699ebd050734aeb64]) successfully deleted
I20260924 19:12:29.466285 13692 ts_tablet_manager.cc:1939] T 8981d15378cc4c4e8054dfe3dc4230cd P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting consensus metadata
I20260924 19:12:29.467448 13082 catalog_manager.cc:4841] TS 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297): tablet 8981d15378cc4c4e8054dfe3dc4230cd (table test-table [id=0b95087e93b1405699ebd050734aeb64]) successfully deleted
I20260924 19:12:29.468247 13693 ts_tablet_manager.cc:1918] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:12:29.468385 13693 log.cc:1196] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/8981d15378cc4c4e8054dfe3dc4230cd
I20260924 19:12:29.468838 13693 ts_tablet_manager.cc:1939] T 8981d15378cc4c4e8054dfe3dc4230cd P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting consensus metadata
I20260924 19:12:29.469906 13082 catalog_manager.cc:4841] TS e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277): tablet 8981d15378cc4c4e8054dfe3dc4230cd (table test-table [id=0b95087e93b1405699ebd050734aeb64]) successfully deleted
Sep 24 19:12:29 dist-test-slave-0lgr krb5kdc[13050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:12:29 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:29.505118 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:29.488807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.11.18.126:46147)
0924 19:12:29.489523 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:29.489527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:29.489551 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:12:29.489610 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:29.489614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:29.489685 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:29.489796 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:29.491427 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:29.492850 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:29.494426 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:29.494841 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:29.494965 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:29.499330 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:29.499335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:29.499339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:29.499376 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:29.502316 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:29.503098 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:29.503103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:29.503105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:29.503232 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:29.503650 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:29.503655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:29.503658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:29.504046 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:29.504261 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:29.504602 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 19:12:29.504779 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":360,"threads_started":1}
I20260924 19:12:29.509160 13095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:12:29.509886 13095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:29.516788 13233 tablet_service.cc:1467] Processing CreateTablet for tablet 08f6eef0bc00424cabc7d3f87d940c1d (DEFAULT_TABLE table=test-table [id=655bf3636d3643d9bc1c0ddc7bf3923e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:29.516872 13373 tablet_service.cc:1467] Processing CreateTablet for tablet 08f6eef0bc00424cabc7d3f87d940c1d (DEFAULT_TABLE table=test-table [id=655bf3636d3643d9bc1c0ddc7bf3923e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:29.517097 13511 tablet_service.cc:1467] Processing CreateTablet for tablet 08f6eef0bc00424cabc7d3f87d940c1d (DEFAULT_TABLE table=test-table [id=655bf3636d3643d9bc1c0ddc7bf3923e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:29.517268 13373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f6eef0bc00424cabc7d3f87d940c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:29.517268 13233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f6eef0bc00424cabc7d3f87d940c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:29.517570 13511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f6eef0bc00424cabc7d3f87d940c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:29.524452 13703 tablet_bootstrap.cc:492] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Bootstrap starting.
I20260924 19:12:29.525271 13702 tablet_bootstrap.cc:492] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: Bootstrap starting.
I20260924 19:12:29.526829 13703 tablet_bootstrap.cc:654] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:29.528151 13702 tablet_bootstrap.cc:654] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:29.528488 13704 tablet_bootstrap.cc:492] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: Bootstrap starting.
I20260924 19:12:29.529860 13703 tablet_bootstrap.cc:492] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: No bootstrap required, opened a new log
I20260924 19:12:29.530017 13703 ts_tablet_manager.cc:1397] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 19:12:29.530133 13704 tablet_bootstrap.cc:654] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:29.530405 13702 tablet_bootstrap.cc:492] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: No bootstrap required, opened a new log
I20260924 19:12:29.530591 13702 ts_tablet_manager.cc:1397] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 19:12:29.531167 13703 raft_consensus.cc:348] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.531383 13702 raft_consensus.cc:348] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.531512 13703 raft_consensus.cc:374] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:29.531630 13703 raft_consensus.cc:729] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, State: Initialized, Role: FOLLOWER
I20260924 19:12:29.532044 13702 raft_consensus.cc:374] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:29.532156 13702 raft_consensus.cc:729] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6326fc66bbe4eae9b10a29ee95f9a4f, State: Initialized, Role: FOLLOWER
I20260924 19:12:29.532210 13703 consensus_queue.cc:260] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.532967 13703 ts_tablet_manager.cc:1428] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:12:29.533258 13702 consensus_queue.cc:260] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.533595 13686 raft_consensus.cc:482] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:29.533767 13686 raft_consensus.cc:504] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.534618 13686 leader_election.cc:288] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897), 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297)
I20260924 19:12:29.535148 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f6eef0bc00424cabc7d3f87d940c1d" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" is_pre_election: true
I20260924 19:12:29.535601 13253 raft_consensus.cc:2454] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 0.
I20260924 19:12:29.536302 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f6eef0bc00424cabc7d3f87d940c1d" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" is_pre_election: true
I20260924 19:12:29.536689 13466 leader_election.cc:302] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, e6326fc66bbe4eae9b10a29ee95f9a4f; no voters:
W20260924 19:12:29.537163 13466 leader_election.cc:341] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:12:29.537642 13686 raft_consensus.cc:2781] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:29.537760 13686 raft_consensus.cc:482] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:29.537854 13686 raft_consensus.cc:3037] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:29.538653 13704 tablet_bootstrap.cc:492] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: No bootstrap required, opened a new log
I20260924 19:12:29.538826 13704 ts_tablet_manager.cc:1397] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260924 19:12:29.539572 13704 raft_consensus.cc:348] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.539747 13704 raft_consensus.cc:374] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:29.539851 13704 raft_consensus.cc:729] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1ffbee5cd348a2bce5680e18d129e6, State: Initialized, Role: FOLLOWER
I20260924 19:12:29.540135 13704 consensus_queue.cc:260] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.540423 13686 raft_consensus.cc:504] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.541014 13704 ts_tablet_manager.cc:1428] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:12:29.541072 13686 leader_election.cc:288] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 election: Requested vote from peers 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897), 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297)
I20260924 19:12:29.541641 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f6eef0bc00424cabc7d3f87d940c1d" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6"
I20260924 19:12:29.541857 13398 raft_consensus.cc:3037] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:29.541895 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f6eef0bc00424cabc7d3f87d940c1d" candidate_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560"
I20260924 19:12:29.542140 13253 raft_consensus.cc:3037] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:29.544704 13398 raft_consensus.cc:2454] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 1.
I20260924 19:12:29.544867 13253 raft_consensus.cc:2454] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6326fc66bbe4eae9b10a29ee95f9a4f in term 1.
I20260924 19:12:29.545190 13466 leader_election.cc:302] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1ffbee5cd348a2bce5680e18d129e6, e6326fc66bbe4eae9b10a29ee95f9a4f; no voters:
I20260924 19:12:29.545528 13686 raft_consensus.cc:2781] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:29.545763 13686 raft_consensus.cc:686] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 LEADER]: Becoming Leader. State: Replica: e6326fc66bbe4eae9b10a29ee95f9a4f, State: Running, Role: LEADER
I20260924 19:12:29.545759 13702 ts_tablet_manager.cc:1428] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20260924 19:12:29.546180 13686 consensus_queue.cc:237] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.548835 13095 catalog_manager.cc:5495] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f reported cstate change: term changed from 0 to 1, leader changed from <none> to e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67). New cstate: current_term: 1 leader_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } health_report { overall_health: HEALTHY } } }
I20260924 19:12:29.572755 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:29.567465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.11.18.67:40277)
0924 19:12:29.568100 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:29.568104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:29.568122 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:12:29.568176 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:29.568179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:29.568260 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:29.568371 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:29.568829 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:29.569676 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:29.571012 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:29.571429 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:29.571554 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:29.571999 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:29.572110 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:29.572209 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:29.572355 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:12:29.572431 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":348,"threads_started":1}
I20260924 19:12:29.582994 13398 raft_consensus.cc:1261] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Refusing update from remote peer e6326fc66bbe4eae9b10a29ee95f9a4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:29.583029 13253 raft_consensus.cc:1261] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Refusing update from remote peer e6326fc66bbe4eae9b10a29ee95f9a4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:29.583642 13686 consensus_queue.cc:1035] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:29.583926 13624 consensus_queue.cc:1035] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:29.633682 13094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:29.633927 13094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:29.637214 13094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: Sending DeleteTablet for 3 replicas of tablet 08f6eef0bc00424cabc7d3f87d940c1d
I20260924 19:12:29.638430 13233 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f6eef0bc00424cabc7d3f87d940c1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:29 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:41880
I20260924 19:12:29.638612 13373 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f6eef0bc00424cabc7d3f87d940c1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:29 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:46644
I20260924 19:12:29.638667 13512 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f6eef0bc00424cabc7d3f87d940c1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:29 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:52050
I20260924 19:12:29.638767 13691 tablet_replica.cc:330] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: stopping tablet replica
I20260924 19:12:29.638900 13692 tablet_replica.cc:330] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: stopping tablet replica
I20260924 19:12:29.638972 13691 raft_consensus.cc:2229] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:29.639087 13692 raft_consensus.cc:2229] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:29.639178 13691 raft_consensus.cc:2258] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:29.639281 13692 raft_consensus.cc:2258] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:29.639441 13693 tablet_replica.cc:330] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: stopping tablet replica
I20260924 19:12:29.639663 13693 raft_consensus.cc:2229] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:12:29.640020 13693 raft_consensus.cc:2258] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:29.639787 13094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:12:29.640405 13691 ts_tablet_manager.cc:1905] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:29.640497 13692 ts_tablet_manager.cc:1905] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:12:29.640585 13094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:29.642115 13693 ts_tablet_manager.cc:1905] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:29.647626 13691 ts_tablet_manager.cc:1918] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:29.647800 13691 log.cc:1196] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/08f6eef0bc00424cabc7d3f87d940c1d
I20260924 19:12:29.648274 13691 ts_tablet_manager.cc:1939] T 08f6eef0bc00424cabc7d3f87d940c1d P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting consensus metadata
I20260924 19:12:29.649873 13373 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd1229c02b540588d6c53babb996e3e (DEFAULT_TABLE table=test-table [id=1c2c66a5b99f48ceb2f71cfc015f6e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:29.650245 13233 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd1229c02b540588d6c53babb996e3e (DEFAULT_TABLE table=test-table [id=1c2c66a5b99f48ceb2f71cfc015f6e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:29.650547 13082 catalog_manager.cc:4841] TS 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897): tablet 08f6eef0bc00424cabc7d3f87d940c1d (table test-table [id=655bf3636d3643d9bc1c0ddc7bf3923e]) successfully deleted
I20260924 19:12:29.650750 13233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd1229c02b540588d6c53babb996e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:29.651137 13373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd1229c02b540588d6c53babb996e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:29.649519 13512 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd1229c02b540588d6c53babb996e3e (DEFAULT_TABLE table=test-table [id=1c2c66a5b99f48ceb2f71cfc015f6e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:29.651571 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd1229c02b540588d6c53babb996e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:29.653959 13692 ts_tablet_manager.cc:1918] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:29.650266 13693 ts_tablet_manager.cc:1918] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:12:29.654630 13693 log.cc:1196] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/08f6eef0bc00424cabc7d3f87d940c1d
I20260924 19:12:29.655257 13693 ts_tablet_manager.cc:1939] T 08f6eef0bc00424cabc7d3f87d940c1d P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting consensus metadata
I20260924 19:12:29.655480 13692 log.cc:1196] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/08f6eef0bc00424cabc7d3f87d940c1d
I20260924 19:12:29.657090 13082 catalog_manager.cc:4841] TS e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277): tablet 08f6eef0bc00424cabc7d3f87d940c1d (table test-table [id=655bf3636d3643d9bc1c0ddc7bf3923e]) successfully deleted
I20260924 19:12:29.657285 13692 ts_tablet_manager.cc:1939] T 08f6eef0bc00424cabc7d3f87d940c1d P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting consensus metadata
I20260924 19:12:29.658392 13082 catalog_manager.cc:4841] TS 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297): tablet 08f6eef0bc00424cabc7d3f87d940c1d (table test-table [id=655bf3636d3643d9bc1c0ddc7bf3923e]) successfully deleted
I20260924 19:12:29.659262 13703 tablet_bootstrap.cc:492] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Bootstrap starting.
I20260924 19:12:29.660192 13704 tablet_bootstrap.cc:492] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: Bootstrap starting.
I20260924 19:12:29.660583 13702 tablet_bootstrap.cc:492] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: Bootstrap starting.
I20260924 19:12:29.660959 13703 tablet_bootstrap.cc:654] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:29.661892 13704 tablet_bootstrap.cc:654] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:29.662724 13702 tablet_bootstrap.cc:654] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:29.663623 13703 tablet_bootstrap.cc:492] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: No bootstrap required, opened a new log
I20260924 19:12:29.663848 13703 ts_tablet_manager.cc:1397] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 19:12:29.664598 13704 tablet_bootstrap.cc:492] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: No bootstrap required, opened a new log
I20260924 19:12:29.664654 13703 raft_consensus.cc:348] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.664844 13704 ts_tablet_manager.cc:1397] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 19:12:29.664913 13703 raft_consensus.cc:374] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:29.664927 13702 tablet_bootstrap.cc:492] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: No bootstrap required, opened a new log
I20260924 19:12:29.665004 13703 raft_consensus.cc:729] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, State: Initialized, Role: FOLLOWER
I20260924 19:12:29.665124 13702 ts_tablet_manager.cc:1397] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260924 19:12:29.665251 13703 consensus_queue.cc:260] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.665786 13704 raft_consensus.cc:348] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.666175 13703 ts_tablet_manager.cc:1428] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:12:29.666237 13704 raft_consensus.cc:374] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:29.666322 13704 raft_consensus.cc:729] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1ffbee5cd348a2bce5680e18d129e6, State: Initialized, Role: FOLLOWER
I20260924 19:12:29.666571 13704 consensus_queue.cc:260] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.666774 13702 raft_consensus.cc:348] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.667407 13702 raft_consensus.cc:374] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:29.667505 13702 raft_consensus.cc:729] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6326fc66bbe4eae9b10a29ee95f9a4f, State: Initialized, Role: FOLLOWER
I20260924 19:12:29.668144 13704 ts_tablet_manager.cc:1428] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:29.669302 13702 consensus_queue.cc:260] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.670282 13702 ts_tablet_manager.cc:1428] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260924 19:12:29.785439 13668 raft_consensus.cc:482] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:29.785723 13668 raft_consensus.cc:504] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.786506 13668 leader_election.cc:288] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297), e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277)
I20260924 19:12:29.787242 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd1229c02b540588d6c53babb996e3e" candidate_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" is_pre_election: true
I20260924 19:12:29.787242 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd1229c02b540588d6c53babb996e3e" candidate_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" is_pre_election: true
I20260924 19:12:29.787526 13398 raft_consensus.cc:2454] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bbcf3a78ebe4c1da84e9ef2eb7c4560 in term 0.
I20260924 19:12:29.787526 13537 raft_consensus.cc:2454] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bbcf3a78ebe4c1da84e9ef2eb7c4560 in term 0.
I20260924 19:12:29.788045 13186 leader_election.cc:302] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1ffbee5cd348a2bce5680e18d129e6, 9bbcf3a78ebe4c1da84e9ef2eb7c4560; no voters:
I20260924 19:12:29.788329 13668 raft_consensus.cc:2781] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:29.788457 13668 raft_consensus.cc:482] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:29.788568 13668 raft_consensus.cc:3037] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:29.791412 13668 raft_consensus.cc:504] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.792073 13668 leader_election.cc:288] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [CANDIDATE]: Term 1 election: Requested vote from peers 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297), e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277)
I20260924 19:12:29.792654 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd1229c02b540588d6c53babb996e3e" candidate_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1ffbee5cd348a2bce5680e18d129e6"
I20260924 19:12:29.792722 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd1229c02b540588d6c53babb996e3e" candidate_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f"
I20260924 19:12:29.792902 13398 raft_consensus.cc:3037] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:29.792925 13537 raft_consensus.cc:3037] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:29.795182 13537 raft_consensus.cc:2454] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bbcf3a78ebe4c1da84e9ef2eb7c4560 in term 1.
I20260924 19:12:29.795331 13398 raft_consensus.cc:2454] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bbcf3a78ebe4c1da84e9ef2eb7c4560 in term 1.
I20260924 19:12:29.795714 13186 leader_election.cc:302] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, e6326fc66bbe4eae9b10a29ee95f9a4f; no voters:
I20260924 19:12:29.796097 13668 raft_consensus.cc:2781] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:29.796269 13668 raft_consensus.cc:686] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 LEADER]: Becoming Leader. State: Replica: 9bbcf3a78ebe4c1da84e9ef2eb7c4560, State: Running, Role: LEADER
I20260924 19:12:29.796583 13668 consensus_queue.cc:237] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } }
I20260924 19:12:29.799566 13094 catalog_manager.cc:5495] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65). New cstate: current_term: 1 leader_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbcf3a78ebe4c1da84e9ef2eb7c4560" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 } health_report { overall_health: UNKNOWN } } }
I20260924 19:12:29.840137 13719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:29.834697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.11.18.65:44897)
0924 19:12:29.835296 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:29.835301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:29.835317 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:12:29.835366 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:29.835369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:29.835426 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:29.835537 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:29.835960 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:29.836722 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:29.838120 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:29.838744 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:29.838890 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:29.839328 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:29.839440 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:29.839539 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:29.839684 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 19:12:29.839759 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":298,"threads_started":1}
I20260924 19:12:29.849706 13537 raft_consensus.cc:1261] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Refusing update from remote peer 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:29.849722 13398 raft_consensus.cc:1261] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Refusing update from remote peer 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:29.850303 13668 consensus_queue.cc:1035] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6326fc66bbe4eae9b10a29ee95f9a4f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:29.850713 13681 consensus_queue.cc:1035] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f1ffbee5cd348a2bce5680e18d129e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:12:29.870729 13673 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:12:31.478670 13095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:31.479043 13095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:31.483618 13095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8d992880fa54e18ab2acd3ffbf0b8ba: Sending DeleteTablet for 3 replicas of tablet 4dd1229c02b540588d6c53babb996e3e
I20260924 19:12:31.484740 13233 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd1229c02b540588d6c53babb996e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:31 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:41880
I20260924 19:12:31.484738 13373 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd1229c02b540588d6c53babb996e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:31 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:46644
I20260924 19:12:31.484830 13511 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd1229c02b540588d6c53babb996e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:31 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:52050
I20260924 19:12:31.488559 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13167
I20260924 19:12:31.500703 13747 tablet_replica.cc:330] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: stopping tablet replica
I20260924 19:12:31.501046 13747 raft_consensus.cc:2229] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:12:31.502137 13747 raft_consensus.cc:2258] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:31.503846 13747 ts_tablet_manager.cc:1905] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:31.509994 13749 tablet_replica.cc:330] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: stopping tablet replica
I20260924 19:12:31.510295 13749 raft_consensus.cc:2229] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:31.510686 13749 raft_consensus.cc:2258] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:31.512564 13749 ts_tablet_manager.cc:1905] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:31.515434 13747 ts_tablet_manager.cc:1918] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:12:31.515620 13747 log.cc:1196] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/4dd1229c02b540588d6c53babb996e3e
I20260924 19:12:31.516271 13747 ts_tablet_manager.cc:1939] T 4dd1229c02b540588d6c53babb996e3e P 9bbcf3a78ebe4c1da84e9ef2eb7c4560: Deleting consensus metadata
I20260924 19:12:31.517940 13082 catalog_manager.cc:4841] TS 9bbcf3a78ebe4c1da84e9ef2eb7c4560 (127.11.18.65:44897): tablet 4dd1229c02b540588d6c53babb996e3e (table test-table [id=1c2c66a5b99f48ceb2f71cfc015f6e83]) successfully deleted
I20260924 19:12:31.519181 13748 tablet_replica.cc:330] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: stopping tablet replica
I20260924 19:12:31.519469 13748 raft_consensus.cc:2229] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:31.519830 13748 raft_consensus.cc:2258] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:31.521013 13748 ts_tablet_manager.cc:1905] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:31.522272 13749 ts_tablet_manager.cc:1918] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:12:31.522428 13749 log.cc:1196] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/4dd1229c02b540588d6c53babb996e3e
I20260924 19:12:31.523077 13749 ts_tablet_manager.cc:1939] T 4dd1229c02b540588d6c53babb996e3e P e6326fc66bbe4eae9b10a29ee95f9a4f: Deleting consensus metadata
I20260924 19:12:31.524151 13082 catalog_manager.cc:4841] TS e6326fc66bbe4eae9b10a29ee95f9a4f (127.11.18.67:40277): tablet 4dd1229c02b540588d6c53babb996e3e (table test-table [id=1c2c66a5b99f48ceb2f71cfc015f6e83]) successfully deleted
I20260924 19:12:31.526981 13748 ts_tablet_manager.cc:1918] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:12:31.527092 13748 log.cc:1196] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/4dd1229c02b540588d6c53babb996e3e
I20260924 19:12:31.527624 13748 ts_tablet_manager.cc:1939] T 4dd1229c02b540588d6c53babb996e3e P 0f1ffbee5cd348a2bce5680e18d129e6: Deleting consensus metadata
I20260924 19:12:31.528644 13082 catalog_manager.cc:4841] TS 0f1ffbee5cd348a2bce5680e18d129e6 (127.11.18.66:42297): tablet 4dd1229c02b540588d6c53babb996e3e (table test-table [id=1c2c66a5b99f48ceb2f71cfc015f6e83]) successfully deleted
Sep 24 19:12:31 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:31.543779 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:31.490317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.11.18.65:44897)
0924 19:12:31.518146 (+ 27829us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:31.518156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:31.518829 (+ 673us) server_negotiation.cc:407] Connection header received
0924 19:12:31.519108 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:31.519115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:31.519209 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:31.519341 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:31.521364 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:31.522697 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:31.524663 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:31.525181 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:31.525299 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:31.537181 (+ 11882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:31.537200 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:31.537216 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:31.537264 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:31.540462 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:31.541517 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:31.541524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:31.541527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:31.541674 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:31.542188 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:31.542192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:31.542194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:31.542576 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:31.542817 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:31.543267 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 19:12:31.543461 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27563,"thread_start_us":27283,"threads_started":1}
W20260924 19:12:31.544591 13298 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41900
W20260924 19:12:31.545462 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13167: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:31.545733 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 13167
I20260924 19:12:31.571471 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13307
Sep 24 19:12:31 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:12:31.589582 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:31.572319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.11.18.66:42297)
0924 19:12:31.572861 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:31.572865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:31.572881 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:12:31.572934 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:31.572938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:31.573006 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:31.573099 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:31.574559 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:31.575531 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:31.576848 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:31.577427 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:31.577551 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:31.582441 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:31.582457 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:31.582474 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:31.582510 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:31.586091 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:31.587147 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:31.587156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:31.587159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:31.587359 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:31.587817 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:31.587821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:31.587823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:31.588300 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:31.588546 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:31.588889 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 19:12:31.589140 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":317,"threads_started":1}
W20260924 19:12:31.590428 13438 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46658
W20260924 19:12:31.591053 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13307: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:31.591282 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 13307
I20260924 19:12:31.610467 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13447
Sep 24 19:12:31 dist-test-slave-0lgr krb5kdc[13050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:12:31.628337 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:31.611464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.11.18.67:40277)
0924 19:12:31.611962 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:31.611966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:31.611985 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:12:31.612038 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:31.612041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:31.612106 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:31.612207 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:31.613684 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:31.614508 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:31.616041 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:31.616472 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:31.616584 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:31.621871 (+ 5287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:31.621888 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:31.621913 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:31.621956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:31.625309 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:31.626268 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:31.626277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:31.626279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:31.626408 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:31.626871 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:31.626874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:31.626876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:31.627319 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:31.627541 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:31.627883 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 19:12:31.628072 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":276,"threads_started":1}
W20260924 19:12:31.629120 13577 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52092
W20260924 19:12:31.629791 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13447: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:31.630045 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 13447
I20260924 19:12:31.655633 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13063
I20260924 19:12:31.657009 13135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:31.775488 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 13063
2026-09-24T19:12:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8873 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:31 dist-test-slave-0lgr krb5kdc[13757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:31 dist-test-slave-0lgr krb5kdc[13757](info): set up 2 sockets
Sep 24 19:12:31 dist-test-slave-0lgr krb5kdc[13757](info): commencing operation
krb5kdc: starting...
W20260924 19:12:32.893555 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:33 dist-test-slave-0lgr krb5kdc[13757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:33.094357 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38971
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:33.424266 13770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:33.424580 13770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:33.424755 13770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:33.432379 13770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:33.432514 13770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:33.432641 13770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:33.432827 13770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:33.432899 13770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:33.444240 13770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:38971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:33.445791 13770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:33.447233 13770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:33.460919 13775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:33.461067 13778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:33.461067 13776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:33.461396 13770 server_base.cc:1034] running on GCE node
I20260924 19:12:33.462543 13770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:33.464049 13770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:33.465435 13770 hybrid_clock.cc:648] HybridClock initialized: now 1790277153465360 us; error 44 us; skew 500 ppm
I20260924 19:12:33.465838 13770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:33 dist-test-slave-0lgr krb5kdc[13757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277153, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:33.469259 13770 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:33.470999 13770 webserver.cc:460] Webserver started at http://127.11.18.126:37657/ using document root <none> and password file <none>
I20260924 19:12:33.471751 13770 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:33.471908 13770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:33.472244 13770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:33.475507 13770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "ed13c7a79f50443f9dc21c2a769c08bc"
format_stamp: "Formatted at 2026-09-24 19:12:33 on dist-test-slave-0lgr"
server_key: "379085cc5e0304ca4afbb0d976c507c4"
server_key_iv: "6db7f6c0066b6bf88689c417379eb414"
server_key_version: "encryptionkey@0"
I20260924 19:12:33.476500 13770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "ed13c7a79f50443f9dc21c2a769c08bc"
format_stamp: "Formatted at 2026-09-24 19:12:33 on dist-test-slave-0lgr"
server_key: "379085cc5e0304ca4afbb0d976c507c4"
server_key_iv: "6db7f6c0066b6bf88689c417379eb414"
server_key_version: "encryptionkey@0"
I20260924 19:12:33.482662 13770 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:12:33.486330 13785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:33.487807 13770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:12:33.488019 13770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "ed13c7a79f50443f9dc21c2a769c08bc"
format_stamp: "Formatted at 2026-09-24 19:12:33 on dist-test-slave-0lgr"
server_key: "379085cc5e0304ca4afbb0d976c507c4"
server_key_iv: "6db7f6c0066b6bf88689c417379eb414"
server_key_version: "encryptionkey@0"
I20260924 19:12:33.488234 13770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:33.516690 13770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:33.521301 13770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:33.521659 13770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:33.541543 13770 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:38971
I20260924 19:12:33.541586 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:38971 every 8 connection(s)
I20260924 19:12:33.542955 13770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:33.544710 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 13770
I20260924 19:12:33.544966 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:33.545862 11337 external_mini_cluster.cc:1428] Setting key 1dbaafe674292ee060d19af35cef2dee
I20260924 19:12:33.548261 13837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:33.559756 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc: Bootstrap starting.
Sep 24 19:12:33 dist-test-slave-0lgr krb5kdc[13757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:33.563674 13837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:33.564754 13837 log.cc:824] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:33.568262 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc: No bootstrap required, opened a new log
I20260924 19:12:33.569857 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:33.548329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.11.18.126:38971)
0924 19:12:33.549148 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:33.549160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:33.549204 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:12:33.550965 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:33.550987 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:33.551395 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:33.551821 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:33.553277 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:33.554668 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:33.555982 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:33.556739 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:33.556974 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:33.561544 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:33.561552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:33.561570 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:33.561618 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:33.565057 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:33.565965 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:33.565974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:33.565983 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:33.566208 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:33.566722 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:33.566726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:33.566729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:33.567296 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:33.567568 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:33.568104 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 19:12:33.568656 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":354,"threads_started":1}
I20260924 19:12:33.576220 13837 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13c7a79f50443f9dc21c2a769c08bc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38971 } }
I20260924 19:12:33.576612 13837 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:33.576719 13837 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed13c7a79f50443f9dc21c2a769c08bc, State: Initialized, Role: FOLLOWER
I20260924 19:12:33.577277 13837 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13c7a79f50443f9dc21c2a769c08bc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38971 } }
I20260924 19:12:33.577581 13837 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:33.577734 13837 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:33.577916 13837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:33.580075 13837 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13c7a79f50443f9dc21c2a769c08bc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38971 } }
I20260924 19:12:33.580606 13837 leader_election.cc:302] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed13c7a79f50443f9dc21c2a769c08bc; no voters:
I20260924 19:12:33.581195 13837 leader_election.cc:288] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:33.581405 13842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:33.582108 13842 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [term 1 LEADER]: Becoming Leader. State: Replica: ed13c7a79f50443f9dc21c2a769c08bc, State: Running, Role: LEADER
I20260924 19:12:33.582638 13842 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13c7a79f50443f9dc21c2a769c08bc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38971 } }
I20260924 19:12:33.583331 13837 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:33.585604 13844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed13c7a79f50443f9dc21c2a769c08bc. Latest consensus state: current_term: 1 leader_uuid: "ed13c7a79f50443f9dc21c2a769c08bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13c7a79f50443f9dc21c2a769c08bc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38971 } } }
I20260924 19:12:33.586102 13844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:33.585573 13843 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed13c7a79f50443f9dc21c2a769c08bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13c7a79f50443f9dc21c2a769c08bc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38971 } } }
I20260924 19:12:33.586578 13843 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:33.588189 13850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:33.594018 13850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:33.602860 13850 catalog_manager.cc:1347] Generated new cluster ID: cf35356d63bc41db932b21e4a43be139
I20260924 19:12:33.602991 13850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:33.633687 13850 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:33.635152 13850 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:33.645107 13850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed13c7a79f50443f9dc21c2a769c08bc: Generated new TSK 0
I20260924 19:12:33.646042 13850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:12:33.707094 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38971
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:34.060240 13863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:34.060595 13863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:34.060777 13863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:34.068243 13863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:34.068404 13863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:34.068593 13863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:34.079311 13863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:34.080646 13863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:34.082297 13863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:34.096649 13868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:34.096725 13869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:34.096698 13871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:34.097440 13863 server_base.cc:1034] running on GCE node
I20260924 19:12:34.097994 13863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:34.098979 13863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:34.100219 13863 hybrid_clock.cc:648] HybridClock initialized: now 1790277154100147 us; error 65 us; skew 500 ppm
I20260924 19:12:34.100719 13863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:34 dist-test-slave-0lgr krb5kdc[13757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277154, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:34.104444 13863 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:12:34.106215 13863 webserver.cc:460] Webserver started at http://127.11.18.65:41479/ using document root <none> and password file <none>
I20260924 19:12:34.107136 13863 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:34.107293 13863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:34.107625 13863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:34.110900 13863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "64d078dfe9cc407896d48b5c58d07ac2"
format_stamp: "Formatted at 2026-09-24 19:12:34 on dist-test-slave-0lgr"
server_key: "1dcc33a3c06b4f6e7fc2b4ceb594cf24"
server_key_iv: "37497ca86373a76f5317f7d02916057a"
server_key_version: "encryptionkey@0"
I20260924 19:12:34.111855 13863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "64d078dfe9cc407896d48b5c58d07ac2"
format_stamp: "Formatted at 2026-09-24 19:12:34 on dist-test-slave-0lgr"
server_key: "1dcc33a3c06b4f6e7fc2b4ceb594cf24"
server_key_iv: "37497ca86373a76f5317f7d02916057a"
server_key_version: "encryptionkey@0"
I20260924 19:12:34.117892 13863 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:12:34.121897 13878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:34.123567 13863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:34.123855 13863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "64d078dfe9cc407896d48b5c58d07ac2"
format_stamp: "Formatted at 2026-09-24 19:12:34 on dist-test-slave-0lgr"
server_key: "1dcc33a3c06b4f6e7fc2b4ceb594cf24"
server_key_iv: "37497ca86373a76f5317f7d02916057a"
server_key_version: "encryptionkey@0"
I20260924 19:12:34.124099 13863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:34.152829 13863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:34.158083 13863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:34.158406 13863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:34.159444 13863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:34.160866 13863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:34.160969 13863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:34.161114 13863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:34.161186 13863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:34.197794 13863 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:40279
I20260924 19:12:34.197897 13990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:40279 every 8 connection(s)
I20260924 19:12:34.199168 13863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:34.209244 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 13863
I20260924 19:12:34.209568 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:34.210354 11337 external_mini_cluster.cc:1428] Setting key 37e61989ea41654455e89ee49fbee50e
Sep 24 19:12:34 dist-test-slave-0lgr krb5kdc[13757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277154, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:34.222724 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:34.201926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:52181 (local address 127.11.18.126:38971)
0924 19:12:34.202419 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:34.202425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:34.203482 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 19:12:34.205306 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:34.205310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:34.205650 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:34.205748 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:34.207454 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:34.208269 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:34.210194 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:34.210885 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:34.211084 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:34.215126 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:34.215136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:34.215145 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:34.215198 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:34.218730 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:34.219582 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:34.219589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:34.219593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:34.219727 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:34.220223 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:34.220230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:34.220238 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:34.220814 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:34.221155 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:34.222055 (+ 900us) server_negotiation.cc:299] Negotiation successful
0924 19:12:34.222308 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":225,"threads_started":1}
I20260924 19:12:34.223807 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:34.202534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38971 (local address 127.11.18.65:52181)
0924 19:12:34.203314 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:34.203352 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:34.205025 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:34.206078 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:34.206087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:34.206646 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:34.207261 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:34.207286 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:34.208457 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:34.208462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:34.210029 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:34.210042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:34.211173 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:34.211181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:34.211630 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:34.212639 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:34.212662 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:34.214873 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:34.218918 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:34.218928 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:34.218948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:34.219441 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:34.219860 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:34.219866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:34.219874 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:34.220085 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:34.221123 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:34.221133 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:34.221598 (+ 465us) client_negotiation.cc:769] Sending connection context
0924 19:12:34.221928 (+ 330us) client_negotiation.cc:240] Negotiation successful
0924 19:12:34.222514 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":351,"threads_started":1}
I20260924 19:12:34.225718 13991 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38971
I20260924 19:12:34.226074 13991 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:34.226857 13991 heartbeater.cc:511] Master 127.11.18.126:38971 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:12:34.229580 13802 ts_manager.cc:194] Registered new tserver with Master: 64d078dfe9cc407896d48b5c58d07ac2 (127.11.18.65:40279)
I20260924 19:12:34.231063 13802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:52181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:12:34.258847 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38971
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:34.590587 13997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:34.590956 13997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:34.591106 13997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:34.599964 13997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:34.600080 13997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:34.600255 13997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:34.610432 13997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:34.611773 13997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:34.613288 13997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:34.633021 14003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:34.633373 14002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:34.633665 14005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:34.634284 13997 server_base.cc:1034] running on GCE node
I20260924 19:12:34.634955 13997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:34.636101 13997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:34.637451 13997 hybrid_clock.cc:648] HybridClock initialized: now 1790277154637375 us; error 44 us; skew 500 ppm
I20260924 19:12:34.637876 13997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:34 dist-test-slave-0lgr krb5kdc[13757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277154, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:34.641433 13997 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:12:34.643159 13997 webserver.cc:460] Webserver started at http://127.11.18.66:33773/ using document root <none> and password file <none>
I20260924 19:12:34.643939 13997 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:34.644096 13997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:34.644433 13997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:34.647719 13997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "ad3a641960c842bb814a79f53882a322"
format_stamp: "Formatted at 2026-09-24 19:12:34 on dist-test-slave-0lgr"
server_key: "1a8f887f1ce758248f0ea56814fa0530"
server_key_iv: "80cc6d78ab385975cb157201620e9738"
server_key_version: "encryptionkey@0"
I20260924 19:12:34.648613 13997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "ad3a641960c842bb814a79f53882a322"
format_stamp: "Formatted at 2026-09-24 19:12:34 on dist-test-slave-0lgr"
server_key: "1a8f887f1ce758248f0ea56814fa0530"
server_key_iv: "80cc6d78ab385975cb157201620e9738"
server_key_version: "encryptionkey@0"
I20260924 19:12:34.654399 13997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:12:34.658207 14012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:34.659780 13997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 19:12:34.659997 13997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "ad3a641960c842bb814a79f53882a322"
format_stamp: "Formatted at 2026-09-24 19:12:34 on dist-test-slave-0lgr"
server_key: "1a8f887f1ce758248f0ea56814fa0530"
server_key_iv: "80cc6d78ab385975cb157201620e9738"
server_key_version: "encryptionkey@0"
I20260924 19:12:34.660219 13997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:34.677260 13997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:34.682236 13997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:34.682536 13997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:34.683634 13997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:34.685101 13997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:34.685201 13997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:34.685385 13997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:34.685469 13997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 19:12:34.721516 13997 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:44241
I20260924 19:12:34.721561 14124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:44241 every 8 connection(s)
I20260924 19:12:34.722829 13997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:34.729751 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 13997
I20260924 19:12:34.729951 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:34.730490 11337 external_mini_cluster.cc:1428] Setting key 30a5a25536cd720ea5248f423ed02f1a
Sep 24 19:12:34 dist-test-slave-0lgr krb5kdc[13757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277154, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:34.744964 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:34.725473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:34799 (local address 127.11.18.126:38971)
0924 19:12:34.725974 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:34.725979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:34.726862 (+ 883us) server_negotiation.cc:407] Connection header received
0924 19:12:34.728732 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:34.728738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:34.728829 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:34.728944 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:34.731026 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:34.731943 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:34.733739 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:34.734613 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:34.734757 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:34.738538 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:34.738548 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:34.738551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:34.738583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:34.741441 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:34.742400 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:34.742406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:34.742409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:34.742577 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:34.743054 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:34.743066 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:34.743068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:34.743327 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:34.743495 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:34.744335 (+ 840us) server_negotiation.cc:299] Negotiation successful
0924 19:12:34.744571 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":401,"thread_start_us":245,"threads_started":1}
I20260924 19:12:34.746134 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:34.725963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38971 (local address 127.11.18.66:34799)
0924 19:12:34.726679 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:34.726717 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:34.728407 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:34.729210 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:34.729218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:34.730043 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:34.730833 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:34.730859 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:34.732110 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:34.732116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:34.733590 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:34.733602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:34.734833 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:34.734838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:34.735053 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:34.735763 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:34.735791 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:34.738226 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:34.741608 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:34.741618 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:34.741647 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:34.742235 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:34.742701 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:34.742706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:34.742709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:34.742925 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:34.743469 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:34.743478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:34.743950 (+ 472us) client_negotiation.cc:769] Sending connection context
0924 19:12:34.744235 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 19:12:34.744820 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":294,"threads_started":1}
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:12:34.747768 14125 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38971
I20260924 19:12:34.748102 14125 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:34.748888 14125 heartbeater.cc:511] Master 127.11.18.126:38971 requested a full tablet report, sending...
I20260924 19:12:34.750523 13802 ts_manager.cc:194] Registered new tserver with Master: ad3a641960c842bb814a79f53882a322 (127.11.18.66:44241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:12:34.751670 13802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:34799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:12:34.777827 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38971
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:35.113772 14131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:35.114105 14131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:35.114276 14131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:35.121743 14131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:35.121876 14131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:35.122045 14131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:35.132290 14131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:35.133721 14131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:35.135226 14131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:35.149710 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:35.149660 14136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:35.149734 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:35.150379 14131 server_base.cc:1034] running on GCE node
I20260924 19:12:35.151057 14131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:35.152114 14131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:35.153452 14131 hybrid_clock.cc:648] HybridClock initialized: now 1790277155153366 us; error 42 us; skew 500 ppm
I20260924 19:12:35.153806 14131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:35 dist-test-slave-0lgr krb5kdc[13757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277155, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:35.157277 14131 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:12:35.158840 14131 webserver.cc:460] Webserver started at http://127.11.18.67:43231/ using document root <none> and password file <none>
I20260924 19:12:35.159507 14131 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:35.159606 14131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:35.159860 14131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:35.162879 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "8995a3bba7c64dc4b7c3516b3a5bf94d"
format_stamp: "Formatted at 2026-09-24 19:12:35 on dist-test-slave-0lgr"
server_key: "282460106cba4dfb02a803822d81fb07"
server_key_iv: "87afe20fddd4c32ce1687f930d29c0ca"
server_key_version: "encryptionkey@0"
I20260924 19:12:35.163625 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "8995a3bba7c64dc4b7c3516b3a5bf94d"
format_stamp: "Formatted at 2026-09-24 19:12:35 on dist-test-slave-0lgr"
server_key: "282460106cba4dfb02a803822d81fb07"
server_key_iv: "87afe20fddd4c32ce1687f930d29c0ca"
server_key_version: "encryptionkey@0"
I20260924 19:12:35.168998 14131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:12:35.172647 14146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:35.174199 14131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:35.174376 14131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "8995a3bba7c64dc4b7c3516b3a5bf94d"
format_stamp: "Formatted at 2026-09-24 19:12:35 on dist-test-slave-0lgr"
server_key: "282460106cba4dfb02a803822d81fb07"
server_key_iv: "87afe20fddd4c32ce1687f930d29c0ca"
server_key_version: "encryptionkey@0"
I20260924 19:12:35.174584 14131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:35.194137 14131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:35.199242 14131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:35.199569 14131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:35.200598 14131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:35.202123 14131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:35.202201 14131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:35.202354 14131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:35.202409 14131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:35.234335 13991 heartbeater.cc:503] Master 127.11.18.126:38971 was elected leader, sending a full tablet report...
I20260924 19:12:35.239745 14131 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:33039
I20260924 19:12:35.239784 14258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:33039 every 8 connection(s)
I20260924 19:12:35.241137 14131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:35.247520 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 14131
I20260924 19:12:35.247771 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:35.248406 11337 external_mini_cluster.cc:1428] Setting key 020e4a3a469067d1288229a807abd12d
Sep 24 19:12:35 dist-test-slave-0lgr krb5kdc[13757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277155, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:35.264389 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:35.243968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:37483 (local address 127.11.18.126:38971)
0924 19:12:35.244151 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:35.244155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:35.245408 (+ 1253us) server_negotiation.cc:407] Connection header received
0924 19:12:35.247756 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:35.247765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:35.247883 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:35.247989 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:35.250201 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:35.251235 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:35.253258 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:35.253783 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:35.253912 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:35.257741 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:35.257751 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:35.257759 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:35.257812 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:35.260579 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:35.261478 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:35.261483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:35.261485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:35.261639 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:35.262255 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:35.262259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:35.262261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:35.262538 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:35.262721 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:35.263712 (+ 991us) server_negotiation.cc:299] Negotiation successful
0924 19:12:35.263910 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:12:35.265534 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:35.244421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38971 (local address 127.11.18.67:37483)
0924 19:12:35.245188 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:35.245232 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:35.247395 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:35.248286 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:35.248300 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:35.249119 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:35.250009 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:35.250031 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:35.251410 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:35.251414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:35.253105 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:35.253118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:35.254026 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:35.254032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:35.254241 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:35.255045 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:35.255069 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:35.257503 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:35.260723 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:35.260731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:35.260745 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:35.261227 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:35.261803 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:35.261807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:35.261810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:35.262065 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:35.262701 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:35.262708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:35.263190 (+ 482us) client_negotiation.cc:769] Sending connection context
0924 19:12:35.263521 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 19:12:35.264095 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":312,"threads_started":1}
I20260924 19:12:35.267055 14259 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38971
I20260924 19:12:35.267413 14259 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:35.268252 14259 heartbeater.cc:511] Master 127.11.18.126:38971 requested a full tablet report, sending...
I20260924 19:12:35.269742 13802 ts_manager.cc:194] Registered new tserver with Master: 8995a3bba7c64dc4b7c3516b3a5bf94d (127.11.18.67:33039)
I20260924 19:12:35.270566 13802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:37483
I20260924 19:12:35.277760 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:12:35.308209 14127 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:35.305519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.11.18.126:38971)
0924 19:12:35.305791 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:35.305801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:35.306170 (+ 369us) server_negotiation.cc:407] Connection header received
0924 19:12:35.306395 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:35.306405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:35.306484 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:35.306601 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:35.307931 (+ 1330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39812: BlockingRecv error: recv got EOF from 127.0.0.1:39812 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:12:35.311324 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13863
W20260924 19:12:35.315229 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13863: Not authorized: Client connection negotiation failed: client connection to 127.11.18.65:40279: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:12:35.315496 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 13863
I20260924 19:12:35.331022 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13997
W20260924 19:12:35.333832 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13997: Not authorized: Client connection negotiation failed: client connection to 127.11.18.66:44241: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:12:35.334002 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 13997
I20260924 19:12:35.349258 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14131
W20260924 19:12:35.352495 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14131: Not authorized: Client connection negotiation failed: client connection to 127.11.18.67:33039: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:12:35.352625 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 14131
I20260924 19:12:35.369721 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 13770
I20260924 19:12:35.371308 13832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:35.474823 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 13770
2026-09-24T19:12:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3702 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:35 dist-test-slave-0lgr krb5kdc[14276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:35 dist-test-slave-0lgr krb5kdc[14276](info): set up 2 sockets
Sep 24 19:12:35 dist-test-slave-0lgr krb5kdc[14276](info): commencing operation
krb5kdc: starting...
W20260924 19:12:36.634241 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:36 dist-test-slave-0lgr krb5kdc[14276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:36.846453 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36903
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:36161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:36903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:37.187798 14289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:37.188097 14289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:37.188257 14289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:37.196065 14289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:37.196249 14289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:37.196372 14289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:37.196470 14289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:37.196568 14289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:37.208684 14289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:36161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:36903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:37.210191 14289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:37.211552 14289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:37.226040 14294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:37.226051 14295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:37.226226 14297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:37.226516 14289 server_base.cc:1034] running on GCE node
I20260924 19:12:37.227679 14289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:37.229477 14289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:37.230693 14289 hybrid_clock.cc:648] HybridClock initialized: now 1790277157230626 us; error 61 us; skew 500 ppm
I20260924 19:12:37.231140 14289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:37 dist-test-slave-0lgr krb5kdc[14276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277157, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:37.240953 14289 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:37.243206 14289 webserver.cc:460] Webserver started at http://127.11.18.126:45629/ using document root <none> and password file <none>
I20260924 19:12:37.244015 14289 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:37.244131 14289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:37.244412 14289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:37.247658 14289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "db416212fc9c4e1f98bb44b888bb05d5"
format_stamp: "Formatted at 2026-09-24 19:12:37 on dist-test-slave-0lgr"
server_key: "1440170b4f39a05ac7fb8f59f7c9b332"
server_key_iv: "3b9dcd790bbbea7afb5b316e5236ee86"
server_key_version: "encryptionkey@0"
I20260924 19:12:37.248442 14289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "db416212fc9c4e1f98bb44b888bb05d5"
format_stamp: "Formatted at 2026-09-24 19:12:37 on dist-test-slave-0lgr"
server_key: "1440170b4f39a05ac7fb8f59f7c9b332"
server_key_iv: "3b9dcd790bbbea7afb5b316e5236ee86"
server_key_version: "encryptionkey@0"
I20260924 19:12:37.253976 14289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:12:37.257913 14304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:37.259467 14289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:12:37.259671 14289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "db416212fc9c4e1f98bb44b888bb05d5"
format_stamp: "Formatted at 2026-09-24 19:12:37 on dist-test-slave-0lgr"
server_key: "1440170b4f39a05ac7fb8f59f7c9b332"
server_key_iv: "3b9dcd790bbbea7afb5b316e5236ee86"
server_key_version: "encryptionkey@0"
I20260924 19:12:37.259876 14289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:37.277010 14289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:37.281721 14289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:37.282038 14289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:37.302146 14289 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:36903
I20260924 19:12:37.302196 14355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:36903 every 8 connection(s)
I20260924 19:12:37.303579 14289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:37.307621 14356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:37.307739 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 14289
I20260924 19:12:37.308049 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:37.308706 11337 external_mini_cluster.cc:1428] Setting key 3e6a3d2165138a70edd1a573dde39918
I20260924 19:12:37.322392 14356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5: Bootstrap starting.
Sep 24 19:12:37 dist-test-slave-0lgr krb5kdc[14276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:37.326064 14356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:37.327487 14356 log.cc:824] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:37.331211 14356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5: No bootstrap required, opened a new log
I20260924 19:12:37.337026 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:37.310672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.11.18.126:36903)
0924 19:12:37.311531 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:37.311546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:37.311600 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:12:37.314478 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:37.314511 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:37.315078 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:37.315565 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:37.317244 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:37.319364 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:37.320932 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:37.321987 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:37.322174 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:37.326577 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:37.326589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:37.326615 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:37.326664 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:37.330619 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:37.331675 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:37.331692 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:37.331699 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:37.331993 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:37.332546 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:37.332553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:37.332558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:37.333457 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:37.333816 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:37.334678 (+ 862us) server_negotiation.cc:299] Negotiation successful
0924 19:12:37.335166 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":334,"threads_started":1}
I20260924 19:12:37.341768 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:12:37.342839 14356 raft_consensus.cc:348] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db416212fc9c4e1f98bb44b888bb05d5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36903 } }
I20260924 19:12:37.343209 14356 raft_consensus.cc:374] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:37.343335 14356 raft_consensus.cc:729] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db416212fc9c4e1f98bb44b888bb05d5, State: Initialized, Role: FOLLOWER
I20260924 19:12:37.343912 14356 consensus_queue.cc:260] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db416212fc9c4e1f98bb44b888bb05d5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36903 } }
I20260924 19:12:37.344127 14356 raft_consensus.cc:388] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:37.344252 14356 raft_consensus.cc:482] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:37.344422 14356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:37.346673 14356 raft_consensus.cc:504] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db416212fc9c4e1f98bb44b888bb05d5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36903 } }
I20260924 19:12:37.347223 14356 leader_election.cc:302] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db416212fc9c4e1f98bb44b888bb05d5; no voters:
I20260924 19:12:37.347882 14356 leader_election.cc:288] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:37.347981 14361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:37.350317 14356 sys_catalog.cc:564] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:37.350584 14361 raft_consensus.cc:686] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [term 1 LEADER]: Becoming Leader. State: Replica: db416212fc9c4e1f98bb44b888bb05d5, State: Running, Role: LEADER
I20260924 19:12:37.351110 14361 consensus_queue.cc:237] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db416212fc9c4e1f98bb44b888bb05d5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36903 } }
I20260924 19:12:37.356381 14363 sys_catalog.cc:455] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db416212fc9c4e1f98bb44b888bb05d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db416212fc9c4e1f98bb44b888bb05d5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36903 } } }
I20260924 19:12:37.356774 14363 sys_catalog.cc:458] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:37.357220 14364 sys_catalog.cc:455] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db416212fc9c4e1f98bb44b888bb05d5. Latest consensus state: current_term: 1 leader_uuid: "db416212fc9c4e1f98bb44b888bb05d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db416212fc9c4e1f98bb44b888bb05d5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36903 } } }
I20260924 19:12:37.357550 14364 sys_catalog.cc:458] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:37.363941 14372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:37.369637 14372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:37.380664 14372 catalog_manager.cc:1347] Generated new cluster ID: 33b547f263914aa5ac21d6bf1b8c761e
I20260924 19:12:37.380796 14372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:37.391341 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:37.382967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.11.18.126:36903)
0924 19:12:37.383108 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:37.383113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:37.383130 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:12:37.383180 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:37.383183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:37.383242 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:37.383329 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:37.384466 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:37.385595 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:37.387545 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:37.388204 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:37.388316 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:37.389634 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:37.389641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:37.389646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:37.390497 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:37.390595 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:37.390628 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:37.390730 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:37.390902 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 19:12:37.391000 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:12:37.394068 14372 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:37.395955 14372 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:37.406519 14372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db416212fc9c4e1f98bb44b888bb05d5: Generated new TSK 0
I20260924 19:12:37.407395 14372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:12:37.417541 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14289
I20260924 19:12:37.418866 14351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:37.509177 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 14289
2026-09-24T19:12:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2026 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:37 dist-test-slave-0lgr krb5kdc[14390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:37 dist-test-slave-0lgr krb5kdc[14390](info): set up 2 sockets
Sep 24 19:12:37 dist-test-slave-0lgr krb5kdc[14390](info): commencing operation
krb5kdc: starting...
W20260924 19:12:38.645891 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:38 dist-test-slave-0lgr krb5kdc[14390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:38.854112 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:34071
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:34071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:39.200366 14403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:39.200739 14403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:39.200845 14403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:39.210943 14403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:39.211076 14403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:39.211126 14403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:39.211191 14403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:39.211246 14403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:39.223018 14403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:34071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:34071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:39.224550 14403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:39.226024 14403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:39.240743 14411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:39.240702 14408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:39.241151 14409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:39.241719 14403 server_base.cc:1034] running on GCE node
I20260924 19:12:39.242611 14403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:39.244406 14403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:39.245882 14403 hybrid_clock.cc:648] HybridClock initialized: now 1790277159245777 us; error 90 us; skew 500 ppm
I20260924 19:12:39.246513 14403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:39 dist-test-slave-0lgr krb5kdc[14390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277159, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:39.249953 14403 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:39.251612 14403 webserver.cc:460] Webserver started at http://127.11.18.126:35689/ using document root <none> and password file <none>
I20260924 19:12:39.252372 14403 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:39.252511 14403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:39.252836 14403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:39.255925 14403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "58e3d52e559e40f1827f6a82eac766c8"
format_stamp: "Formatted at 2026-09-24 19:12:39 on dist-test-slave-0lgr"
server_key: "1ea34253cc6cf4b4e607e06753bd1759"
server_key_iv: "e21a6ab3f2be2cbab8182d0bc9be84b9"
server_key_version: "encryptionkey@0"
I20260924 19:12:39.256711 14403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "58e3d52e559e40f1827f6a82eac766c8"
format_stamp: "Formatted at 2026-09-24 19:12:39 on dist-test-slave-0lgr"
server_key: "1ea34253cc6cf4b4e607e06753bd1759"
server_key_iv: "e21a6ab3f2be2cbab8182d0bc9be84b9"
server_key_version: "encryptionkey@0"
I20260924 19:12:39.262351 14403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 19:12:39.266798 14418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:39.268842 14403 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:12:39.269176 14403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "58e3d52e559e40f1827f6a82eac766c8"
format_stamp: "Formatted at 2026-09-24 19:12:39 on dist-test-slave-0lgr"
server_key: "1ea34253cc6cf4b4e607e06753bd1759"
server_key_iv: "e21a6ab3f2be2cbab8182d0bc9be84b9"
server_key_version: "encryptionkey@0"
I20260924 19:12:39.269484 14403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:39.286811 14403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:39.291417 14403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:39.291721 14403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:39.312563 14403 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:34071
I20260924 19:12:39.312601 14469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:34071 every 8 connection(s)
I20260924 19:12:39.314147 14403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:39.314796 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 14403
I20260924 19:12:39.315122 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:39.315987 11337 external_mini_cluster.cc:1428] Setting key 34896879e646de9ecc2dca4d79973d73
I20260924 19:12:39.319005 14470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:12:39 dist-test-slave-0lgr krb5kdc[14390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:39.331867 14470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8: Bootstrap starting.
I20260924 19:12:39.336521 14470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:39.337992 14470 log.cc:824] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:39.341114 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:39.318191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.11.18.126:34071)
0924 19:12:39.319046 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:39.319059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:39.319099 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:12:39.320729 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:39.320752 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:39.321132 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:39.321623 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:39.323069 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:39.325308 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:39.326705 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:39.327776 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:39.328049 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:39.332659 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:39.332668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:39.332684 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:39.332729 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:39.336219 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:39.337092 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:39.337099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:39.337106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:39.337280 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:39.337736 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:39.337742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:39.337745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:39.338375 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:39.338729 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:39.339167 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 19:12:39.339613 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":415,"threads_started":1}
I20260924 19:12:39.341653 14470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8: No bootstrap required, opened a new log
I20260924 19:12:39.351478 14470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e3d52e559e40f1827f6a82eac766c8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34071 } }
I20260924 19:12:39.351979 14470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:39.352090 14470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e3d52e559e40f1827f6a82eac766c8, State: Initialized, Role: FOLLOWER
I20260924 19:12:39.352753 14470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e3d52e559e40f1827f6a82eac766c8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34071 } }
I20260924 19:12:39.353049 14470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:39.353449 14470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:39.353703 14470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:39.356360 14470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e3d52e559e40f1827f6a82eac766c8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34071 } }
I20260924 19:12:39.356989 14470 leader_election.cc:302] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e3d52e559e40f1827f6a82eac766c8; no voters:
I20260924 19:12:39.357923 14470 leader_election.cc:288] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:39.358342 14475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:39.360885 14475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [term 1 LEADER]: Becoming Leader. State: Replica: 58e3d52e559e40f1827f6a82eac766c8, State: Running, Role: LEADER
I20260924 19:12:39.361407 14470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:39.361594 14475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e3d52e559e40f1827f6a82eac766c8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34071 } }
I20260924 19:12:39.365041 14476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58e3d52e559e40f1827f6a82eac766c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e3d52e559e40f1827f6a82eac766c8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34071 } } }
I20260924 19:12:39.365095 14477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e3d52e559e40f1827f6a82eac766c8. Latest consensus state: current_term: 1 leader_uuid: "58e3d52e559e40f1827f6a82eac766c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e3d52e559e40f1827f6a82eac766c8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34071 } } }
I20260924 19:12:39.365449 14476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:39.365510 14477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:39.369671 14485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:39.374682 14485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:39.385218 14485 catalog_manager.cc:1347] Generated new cluster ID: b7570581905442d287f746532c14bccb
I20260924 19:12:39.385380 14485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:39.396898 14485 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:39.399055 14485 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:39.411993 14485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e3d52e559e40f1827f6a82eac766c8: Generated new TSK 0
I20260924 19:12:39.413232 14485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:12:39.470539 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34071
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:39.835058 14496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:39.835364 14496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:39.835515 14496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:39.843254 14496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:39.843367 14496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:39.843554 14496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:39.853763 14496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:39.855101 14496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:39.856479 14496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:39.871377 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:39.871423 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:39.871348 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:39.872639 14496 server_base.cc:1034] running on GCE node
I20260924 19:12:39.873584 14496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:39.874814 14496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:39.876067 14496 hybrid_clock.cc:648] HybridClock initialized: now 1790277159876000 us; error 69 us; skew 500 ppm
I20260924 19:12:39.876434 14496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:39 dist-test-slave-0lgr krb5kdc[14390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277159, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:39.880004 14496 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:12:39.881780 14496 webserver.cc:460] Webserver started at http://127.11.18.65:44377/ using document root <none> and password file <none>
I20260924 19:12:39.882526 14496 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:39.882676 14496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:39.883059 14496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:39.886162 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "3732248f0b424ce7abedc1d62b33e101"
format_stamp: "Formatted at 2026-09-24 19:12:39 on dist-test-slave-0lgr"
server_key: "c2f115540f1c19aa6b4f63a2cce1e6c8"
server_key_iv: "f46f483f476b541abab19285f3392aff"
server_key_version: "encryptionkey@0"
I20260924 19:12:39.886997 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "3732248f0b424ce7abedc1d62b33e101"
format_stamp: "Formatted at 2026-09-24 19:12:39 on dist-test-slave-0lgr"
server_key: "c2f115540f1c19aa6b4f63a2cce1e6c8"
server_key_iv: "f46f483f476b541abab19285f3392aff"
server_key_version: "encryptionkey@0"
I20260924 19:12:39.892266 14496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 19:12:39.896059 14511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:39.897553 14496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:12:39.897753 14496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "3732248f0b424ce7abedc1d62b33e101"
format_stamp: "Formatted at 2026-09-24 19:12:39 on dist-test-slave-0lgr"
server_key: "c2f115540f1c19aa6b4f63a2cce1e6c8"
server_key_iv: "f46f483f476b541abab19285f3392aff"
server_key_version: "encryptionkey@0"
I20260924 19:12:39.898051 14496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:39.931526 14496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:39.936281 14496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:39.936606 14496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:39.937701 14496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:39.939093 14496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:39.939200 14496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:39.939368 14496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:39.939443 14496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:39.976198 14496 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:34551
I20260924 19:12:39.976230 14623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:34551 every 8 connection(s)
I20260924 19:12:39.977718 14496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:39.984633 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 14496
I20260924 19:12:39.984853 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:39.985560 11337 external_mini_cluster.cc:1428] Setting key e8db3f7e2536338041654988e6cbcce2
Sep 24 19:12:39 dist-test-slave-0lgr krb5kdc[14390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277159, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:40.000887 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:39.980436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:34643 (local address 127.11.18.126:34071)
0924 19:12:39.980949 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:39.980954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:39.981741 (+ 787us) server_negotiation.cc:407] Connection header received
0924 19:12:39.983751 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:39.983757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:39.983879 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:39.984012 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:39.985960 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:39.986902 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:39.988738 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:39.989167 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:39.989318 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:39.993070 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:39.993075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:39.993079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:39.993113 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:39.996869 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:39.997890 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:39.997899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:39.997907 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:39.998067 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:39.998593 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:39.998602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:39.998609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:39.999046 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:39.999209 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:40.000032 (+ 823us) server_negotiation.cc:299] Negotiation successful
0924 19:12:40.000323 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
I20260924 19:12:40.001927 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:39.980818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34071 (local address 127.11.18.65:34643)
0924 19:12:39.981592 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:39.981630 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:39.983426 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:39.984331 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:39.984341 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:39.985024 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:39.985771 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:39.985792 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:39.987072 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:39.987084 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:39.988566 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:39.988579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:39.989460 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:39.989468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:39.989726 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:39.990465 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:39.990495 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:39.992797 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:39.997068 (+ 4271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:39.997082 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:39.997099 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:39.997734 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:39.998212 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:39.998218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:39.998221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:39.998447 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:39.999192 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:39.999200 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:39.999557 (+ 357us) client_negotiation.cc:769] Sending connection context
0924 19:12:39.999862 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 19:12:40.000288 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":262,"threads_started":1}
I20260924 19:12:40.003722 14624 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34071
I20260924 19:12:40.004101 14624 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:40.005122 14624 heartbeater.cc:511] Master 127.11.18.126:34071 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:12:40.007689 14435 ts_manager.cc:194] Registered new tserver with Master: 3732248f0b424ce7abedc1d62b33e101 (127.11.18.65:34551)
I20260924 19:12:40.009194 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:34643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:12:40.033912 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34071
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:40.375496 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:40.375788 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:40.375891 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:40.383348 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:40.383472 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:40.383589 14630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:40.393500 14630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:40.394984 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:40.396356 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:40.410439 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:40.410409 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:40.410359 14638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:40.415334 14630 server_base.cc:1034] running on GCE node
I20260924 19:12:40.416117 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:40.417160 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:40.418429 14630 hybrid_clock.cc:648] HybridClock initialized: now 1790277160418369 us; error 57 us; skew 500 ppm
I20260924 19:12:40.418797 14630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:40 dist-test-slave-0lgr krb5kdc[14390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277160, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:40.422362 14630 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:12:40.424362 14630 webserver.cc:460] Webserver started at http://127.11.18.66:44355/ using document root <none> and password file <none>
I20260924 19:12:40.425128 14630 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:40.425284 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:40.425722 14630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:40.428988 14630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "c589598bfc204111a122e33473aa0f50"
format_stamp: "Formatted at 2026-09-24 19:12:40 on dist-test-slave-0lgr"
server_key: "0cd050bb8a1a42e5bc87705984d0798c"
server_key_iv: "897b914b92635a8d921d330b01a2c97e"
server_key_version: "encryptionkey@0"
I20260924 19:12:40.429895 14630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "c589598bfc204111a122e33473aa0f50"
format_stamp: "Formatted at 2026-09-24 19:12:40 on dist-test-slave-0lgr"
server_key: "0cd050bb8a1a42e5bc87705984d0798c"
server_key_iv: "897b914b92635a8d921d330b01a2c97e"
server_key_version: "encryptionkey@0"
I20260924 19:12:40.435582 14630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:12:40.439327 14645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:40.441119 14630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:12:40.441362 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "c589598bfc204111a122e33473aa0f50"
format_stamp: "Formatted at 2026-09-24 19:12:40 on dist-test-slave-0lgr"
server_key: "0cd050bb8a1a42e5bc87705984d0798c"
server_key_iv: "897b914b92635a8d921d330b01a2c97e"
server_key_version: "encryptionkey@0"
I20260924 19:12:40.441607 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:40.472155 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:40.477149 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:40.477597 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:40.478673 14630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:40.480530 14630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:40.480646 14630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:40.480821 14630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:40.480911 14630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:40.518028 14630 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:37719
I20260924 19:12:40.518118 14757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:37719 every 8 connection(s)
I20260924 19:12:40.519377 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:40.524657 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 14630
I20260924 19:12:40.524857 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:40.525458 11337 external_mini_cluster.cc:1428] Setting key 26fa7a91a03068cf96ad5a73aefa53a6
Sep 24 19:12:40 dist-test-slave-0lgr krb5kdc[14390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277160, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:40.543869 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:40.522284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:55103 (local address 127.11.18.126:34071)
0924 19:12:40.522801 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:40.522805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:40.523888 (+ 1083us) server_negotiation.cc:407] Connection header received
0924 19:12:40.526003 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:40.526009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:40.526104 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:40.526230 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:40.527904 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:40.529049 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:40.530790 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:40.531606 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:40.531774 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:40.536120 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:40.536130 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:40.536138 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:40.536187 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:40.539930 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:40.540798 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:40.540807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:40.540814 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:40.540995 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:40.541658 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:40.541666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:40.541672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:40.542104 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:40.542295 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:40.543153 (+ 858us) server_negotiation.cc:299] Negotiation successful
0924 19:12:40.543425 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":277,"threads_started":1}
I20260924 19:12:40.544878 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:40.522912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34071 (local address 127.11.18.66:55103)
0924 19:12:40.523722 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:40.523761 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:40.525718 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:40.526527 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:40.526536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:40.527054 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:40.527729 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:40.527748 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:40.529221 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:40.529225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:40.530639 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:40.530651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:40.531863 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:40.531868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:40.532090 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:40.532924 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:40.532961 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:40.535850 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:40.540148 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:40.540156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:40.540171 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:40.540650 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:40.541163 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:40.541171 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:40.541178 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:40.541515 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:40.542292 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:40.542300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:40.542717 (+ 417us) client_negotiation.cc:769] Sending connection context
0924 19:12:40.543005 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 19:12:40.543484 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":335,"threads_started":1}
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:12:40.546867 14758 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34071
I20260924 19:12:40.547204 14758 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:40.547966 14758 heartbeater.cc:511] Master 127.11.18.126:34071 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:12:40.549811 14435 ts_manager.cc:194] Registered new tserver with Master: c589598bfc204111a122e33473aa0f50 (127.11.18.66:37719)
I20260924 19:12:40.551213 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:55103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:12:40.578039 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34071
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:40.928262 14764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:40.928567 14764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:40.928732 14764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:40.936111 14764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:40.936230 14764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:40.936466 14764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:40.946699 14764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:40.948028 14764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:40.949402 14764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:40.964094 14769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:40.964044 14770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:40.964365 14772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:40.965004 14764 server_base.cc:1034] running on GCE node
I20260924 19:12:40.965768 14764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:40.966928 14764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:40.968147 14764 hybrid_clock.cc:648] HybridClock initialized: now 1790277160968097 us; error 49 us; skew 500 ppm
I20260924 19:12:40.968552 14764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:40 dist-test-slave-0lgr krb5kdc[14390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277160, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:40.972158 14764 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:12:40.974284 14764 webserver.cc:460] Webserver started at http://127.11.18.67:36505/ using document root <none> and password file <none>
I20260924 19:12:40.975106 14764 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:40.975250 14764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:40.975623 14764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:40.979019 14764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "24f38e750c74495d9fbdaef2b3613fdc"
format_stamp: "Formatted at 2026-09-24 19:12:40 on dist-test-slave-0lgr"
server_key: "2d5d7421329207558b89bf85df41e784"
server_key_iv: "67b4e394e53714ca37316e42305350f4"
server_key_version: "encryptionkey@0"
I20260924 19:12:40.979907 14764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "24f38e750c74495d9fbdaef2b3613fdc"
format_stamp: "Formatted at 2026-09-24 19:12:40 on dist-test-slave-0lgr"
server_key: "2d5d7421329207558b89bf85df41e784"
server_key_iv: "67b4e394e53714ca37316e42305350f4"
server_key_version: "encryptionkey@0"
I20260924 19:12:40.985651 14764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:12:40.989246 14779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:40.990978 14764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:40.991170 14764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "24f38e750c74495d9fbdaef2b3613fdc"
format_stamp: "Formatted at 2026-09-24 19:12:40 on dist-test-slave-0lgr"
server_key: "2d5d7421329207558b89bf85df41e784"
server_key_iv: "67b4e394e53714ca37316e42305350f4"
server_key_version: "encryptionkey@0"
I20260924 19:12:40.991580 14764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:41.012245 14624 heartbeater.cc:503] Master 127.11.18.126:34071 was elected leader, sending a full tablet report...
I20260924 19:12:41.012930 14764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:41.017721 14764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:41.018024 14764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:41.019006 14764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:41.020321 14764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:41.020397 14764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:41.020529 14764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:41.020581 14764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:41.055560 14764 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:35811
I20260924 19:12:41.055629 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:35811 every 8 connection(s)
I20260924 19:12:41.056983 14764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:41.058741 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 14764
I20260924 19:12:41.059010 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:41.059487 11337 external_mini_cluster.cc:1428] Setting key 07775e0b18b82d7fa1a395aff56bcdae
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277160, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:41.080308 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:41.060123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:46967 (local address 127.11.18.126:34071)
0924 19:12:41.060884 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:41.060890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:41.062083 (+ 1193us) server_negotiation.cc:407] Connection header received
0924 19:12:41.064436 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:41.064440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:41.064537 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:41.064645 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:41.066469 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.067348 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.069038 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.069656 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.069794 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:41.073840 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:41.073846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:41.073849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:41.073885 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:41.076992 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.077880 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.077888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.077893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.078021 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.078476 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.078481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.078485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.078774 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:41.078944 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:41.079688 (+ 744us) server_negotiation.cc:299] Negotiation successful
0924 19:12:41.079909 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":472,"threads_started":1}
I20260924 19:12:41.081310 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:41.060923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34071 (local address 127.11.18.67:46967)
0924 19:12:41.061915 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:41.061970 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:41.063950 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:41.064933 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:41.064943 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:41.065685 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:41.066266 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:41.066283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.067516 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.067520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:41.068872 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:41.068885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.069930 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.069936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:41.070151 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:41.070874 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:41.070903 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:41.073588 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:41.077150 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:41.077161 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:41.077182 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:41.077742 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:41.078137 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:41.078142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:41.078146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:41.078353 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:41.078916 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:41.078925 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:41.079320 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 19:12:41.079605 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 19:12:41.080170 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":398,"threads_started":1}
I20260924 19:12:41.082844 14892 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34071
I20260924 19:12:41.083235 14892 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:41.084038 14892 heartbeater.cc:511] Master 127.11.18.126:34071 requested a full tablet report, sending...
I20260924 19:12:41.085636 14435 ts_manager.cc:194] Registered new tserver with Master: 24f38e750c74495d9fbdaef2b3613fdc (127.11.18.67:35811)
I20260924 19:12:41.086505 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:46967
I20260924 19:12:41.088655 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277161, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277161, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:41.140527 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:41.124082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.11.18.126:34071)
0924 19:12:41.124364 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:41.124371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:41.124696 (+ 325us) server_negotiation.cc:407] Connection header received
0924 19:12:41.124813 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:41.124818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:41.124903 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:41.125003 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:41.126698 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.127553 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.129075 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.129618 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.129742 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:41.134276 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:41.134288 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:41.134292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:41.134332 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:41.137309 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.138215 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.138221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.138224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.138380 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.138844 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.138848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.138851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.139206 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:41.139416 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:41.139827 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 19:12:41.140054 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260924 19:12:41.141690 14435 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35984
I20260924 19:12:41.143569 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14496
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277161, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:41.160930 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:41.144625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.11.18.65:34551)
0924 19:12:41.145394 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:41.145401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:41.145425 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:12:41.145480 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:41.145487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:41.145702 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:41.145877 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:41.147271 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.148370 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.149891 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.150529 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.150896 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:41.154838 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:41.154851 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:41.154900 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:41.154945 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:41.157923 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.158679 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.158686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.158689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.158814 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.159274 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.159277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.159279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.159622 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:41.159854 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:41.160237 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 19:12:41.160550 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":400,"threads_started":1}
W20260924 19:12:41.162411 14619 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50300
W20260924 19:12:41.163678 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:41.163939 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 14496
I20260924 19:12:41.179855 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14630
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277161, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:12:41.197484 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:41.180702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.11.18.66:37719)
0924 19:12:41.181243 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:41.181250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:41.181277 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:12:41.181391 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:41.181402 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:41.181628 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:41.181818 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:41.183124 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.184134 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.185561 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.186241 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.186355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:41.190866 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:41.190878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:41.190892 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:41.190929 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:41.193976 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.194787 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.194795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.194797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.194944 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.195449 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.195458 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.195462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.195932 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:41.196201 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:41.196753 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 19:12:41.197040 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":278,"threads_started":1}
W20260924 19:12:41.198904 14753 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44982
W20260924 19:12:41.200107 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14630: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:41.200279 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 14630
I20260924 19:12:41.215788 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14764
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277161, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:12:41.233500 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:41.217003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.11.18.67:35811)
0924 19:12:41.217751 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:41.217759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:41.217789 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:12:41.217851 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:41.217857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:41.218080 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:41.218260 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:41.219606 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.220634 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.222007 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:41.222579 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:41.222737 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:41.227268 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:41.227280 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:41.227295 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:41.227334 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:41.230505 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.231230 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.231238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.231240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.231364 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:41.231760 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:41.231766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:41.231769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:41.232126 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:41.232451 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:41.232769 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 19:12:41.233058 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":434,"threads_started":1}
W20260924 19:12:41.234956 14887 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37214
W20260924 19:12:41.236274 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14764: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:12:41.236577 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 14764
I20260924 19:12:41.254197 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14403
I20260924 19:12:41.255511 14465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:41.368840 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 14403
2026-09-24T19:12:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3867 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14910](info): set up 2 sockets
Sep 24 19:12:41 dist-test-slave-0lgr krb5kdc[14910](info): commencing operation
krb5kdc: starting...
W20260924 19:12:42.534142 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:42 dist-test-slave-0lgr krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:42.750665 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38233
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:38233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:43.094368 14923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:43.094751 14923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:43.094915 14923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:43.102465 14923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:43.102644 14923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:43.102746 14923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:43.102849 14923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:43.102916 14923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:43.112994 14923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:38233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:43.114428 14923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:43.115754 14923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:43.129195 14928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:43.129230 14929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:43.129422 14931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:43.130426 14923 server_base.cc:1034] running on GCE node
I20260924 19:12:43.131413 14923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:43.133112 14923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:43.134409 14923 hybrid_clock.cc:648] HybridClock initialized: now 1790277163134320 us; error 62 us; skew 500 ppm
I20260924 19:12:43.134845 14923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:43 dist-test-slave-0lgr krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277163, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:43.144568 14923 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:43.146546 14923 webserver.cc:460] Webserver started at http://127.11.18.126:34001/ using document root <none> and password file <none>
I20260924 19:12:43.147368 14923 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:43.147543 14923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:43.147897 14923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:43.151443 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "ae5290953fe74b5db67cd02dda87518b"
format_stamp: "Formatted at 2026-09-24 19:12:43 on dist-test-slave-0lgr"
server_key: "d7ce8bd0ef21d79204ae1472c057f0fd"
server_key_iv: "3a37f87595c38893cb560bcc4b07a3c5"
server_key_version: "encryptionkey@0"
I20260924 19:12:43.152356 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "ae5290953fe74b5db67cd02dda87518b"
format_stamp: "Formatted at 2026-09-24 19:12:43 on dist-test-slave-0lgr"
server_key: "d7ce8bd0ef21d79204ae1472c057f0fd"
server_key_iv: "3a37f87595c38893cb560bcc4b07a3c5"
server_key_version: "encryptionkey@0"
I20260924 19:12:43.158284 14923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:12:43.162395 14938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:43.163882 14923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:12:43.164108 14923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "ae5290953fe74b5db67cd02dda87518b"
format_stamp: "Formatted at 2026-09-24 19:12:43 on dist-test-slave-0lgr"
server_key: "d7ce8bd0ef21d79204ae1472c057f0fd"
server_key_iv: "3a37f87595c38893cb560bcc4b07a3c5"
server_key_version: "encryptionkey@0"
I20260924 19:12:43.164348 14923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:43.194331 14923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:43.199321 14923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:43.199651 14923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:43.220619 14923 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:38233
I20260924 19:12:43.220650 14989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:38233 every 8 connection(s)
I20260924 19:12:43.222265 14923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:43.226356 14990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:43.231606 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 14923
I20260924 19:12:43.231871 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:43.232561 11337 external_mini_cluster.cc:1428] Setting key fde4a1fac50bfdb82e843e58ea7ddad7
I20260924 19:12:43.238234 14990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b: Bootstrap starting.
I20260924 19:12:43.242416 14990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:43.243657 14990 log.cc:824] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:43.246826 14990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b: No bootstrap required, opened a new log
Sep 24 19:12:43 dist-test-slave-0lgr krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:43.255667 14990 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5290953fe74b5db67cd02dda87518b" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38233 } }
I20260924 19:12:43.256090 14990 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:43.256276 14990 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5290953fe74b5db67cd02dda87518b, State: Initialized, Role: FOLLOWER
I20260924 19:12:43.257411 14990 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5290953fe74b5db67cd02dda87518b" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38233 } }
I20260924 19:12:43.257802 14995 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:43.258154 14995 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5290953fe74b5db67cd02dda87518b" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38233 } }
I20260924 19:12:43.258817 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:43.234940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.11.18.126:38233)
0924 19:12:43.235930 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:43.235943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:43.235990 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:12:43.237904 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:43.237929 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:43.238403 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:43.238805 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:43.240934 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:43.242534 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:43.244279 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:43.245097 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:43.245358 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:43.250080 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:43.250090 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:43.250108 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:43.250156 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:43.253869 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:43.254765 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:43.254778 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:43.254785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:43.254973 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:43.255391 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:43.255398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:43.255402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:43.256014 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:43.256351 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:43.256842 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 19:12:43.257407 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":510,"threads_started":1}
I20260924 19:12:43.258855 14995 leader_election.cc:302] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5290953fe74b5db67cd02dda87518b; no voters:
I20260924 19:12:43.257773 14990 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:43.259763 14995 leader_election.cc:288] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260924 19:12:43.259852 14990 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:43.260012 14990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:43.262637 14990 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5290953fe74b5db67cd02dda87518b" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38233 } }
I20260924 19:12:43.262914 14990 leader_election.cc:302] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5290953fe74b5db67cd02dda87518b; no voters:
I20260924 19:12:43.262964 14996 raft_consensus.cc:2741] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 19:12:43.263166 14990 leader_election.cc:288] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:43.263834 14997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:43.264093 14997 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [term 1 LEADER]: Becoming Leader. State: Replica: ae5290953fe74b5db67cd02dda87518b, State: Running, Role: LEADER
I20260924 19:12:43.264691 14997 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5290953fe74b5db67cd02dda87518b" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38233 } }
I20260924 19:12:43.265259 14990 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:43.267220 14995 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5290953fe74b5db67cd02dda87518b. Latest consensus state: current_term: 1 leader_uuid: "ae5290953fe74b5db67cd02dda87518b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5290953fe74b5db67cd02dda87518b" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38233 } } }
I20260924 19:12:43.267163 14996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5290953fe74b5db67cd02dda87518b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5290953fe74b5db67cd02dda87518b" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38233 } } }
I20260924 19:12:43.267463 14995 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:43.267490 14996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:43.270594 15000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:43.278682 15000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:43.289997 15000 catalog_manager.cc:1347] Generated new cluster ID: 2c4bb7d904d746cea73e8876c1effefa
I20260924 19:12:43.290138 15000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:43.323951 15000 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:43.325389 15000 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:43.331990 15000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae5290953fe74b5db67cd02dda87518b: Generated new TSK 0
I20260924 19:12:43.333221 15000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:12:43.399039 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38233
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:43.751585 15016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:43.751884 15016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:43.751979 15016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:43.759359 15016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:43.759479 15016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:43.759594 15016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:43.770159 15016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:43.771421 15016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:43.772709 15016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:43.787349 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:43.787444 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:43.787410 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:43.788400 15016 server_base.cc:1034] running on GCE node
I20260924 19:12:43.789146 15016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:43.790274 15016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:43.791595 15016 hybrid_clock.cc:648] HybridClock initialized: now 1790277163791553 us; error 61 us; skew 500 ppm
I20260924 19:12:43.791980 15016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:43 dist-test-slave-0lgr krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277163, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:43.795495 15016 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:12:43.797175 15016 webserver.cc:460] Webserver started at http://127.11.18.65:43279/ using document root <none> and password file <none>
I20260924 19:12:43.798017 15016 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:43.798183 15016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:43.798533 15016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:43.801961 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "6132a855bed44e27bd2d690e4f5a83f0"
format_stamp: "Formatted at 2026-09-24 19:12:43 on dist-test-slave-0lgr"
server_key: "6ea083351c982e49e374c66a395034b7"
server_key_iv: "b2400ab933c4db66ada46d8b31e4d1b0"
server_key_version: "encryptionkey@0"
I20260924 19:12:43.802881 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "6132a855bed44e27bd2d690e4f5a83f0"
format_stamp: "Formatted at 2026-09-24 19:12:43 on dist-test-slave-0lgr"
server_key: "6ea083351c982e49e374c66a395034b7"
server_key_iv: "b2400ab933c4db66ada46d8b31e4d1b0"
server_key_version: "encryptionkey@0"
I20260924 19:12:43.808873 15016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:12:43.813040 15031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:43.814886 15016 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:12:43.815055 15016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "6132a855bed44e27bd2d690e4f5a83f0"
format_stamp: "Formatted at 2026-09-24 19:12:43 on dist-test-slave-0lgr"
server_key: "6ea083351c982e49e374c66a395034b7"
server_key_iv: "b2400ab933c4db66ada46d8b31e4d1b0"
server_key_version: "encryptionkey@0"
I20260924 19:12:43.815243 15016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:43.859383 15016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:43.863945 15016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:43.864214 15016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:43.865170 15016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:43.866611 15016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:43.866683 15016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:43.866809 15016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:43.866870 15016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:43.903144 15016 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:45475
I20260924 19:12:43.903332 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:45475 every 8 connection(s)
I20260924 19:12:43.904491 15016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:43.911770 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 15016
I20260924 19:12:43.912065 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:43.912840 11337 external_mini_cluster.cc:1428] Setting key 448aa91f36b20463c95eec40137a1e9d
Sep 24 19:12:43 dist-test-slave-0lgr krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277163, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:43.928030 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:43.907614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:41121 (local address 127.11.18.126:38233)
0924 19:12:43.908174 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:43.908179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:43.909130 (+ 951us) server_negotiation.cc:407] Connection header received
0924 19:12:43.911252 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:43.911259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:43.911393 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:43.911541 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:43.913475 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:43.914309 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:43.916074 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:43.916529 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:43.916681 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:43.921178 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:43.921193 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:43.921197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:43.921235 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:43.924531 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:43.925464 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:43.925476 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:43.925478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:43.925603 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:43.926100 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:43.926105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:43.926114 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:43.926413 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:43.926600 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:43.927375 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 19:12:43.927621 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":358,"threads_started":1}
I20260924 19:12:43.929075 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:43.908085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38233 (local address 127.11.18.65:41121)
0924 19:12:43.908934 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:43.908980 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:43.910938 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:43.911796 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:43.911806 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:43.912517 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:43.913234 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:43.913257 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:43.914523 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:43.914528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:43.915903 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:43.915916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:43.916799 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:43.916808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:43.917077 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:43.918004 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:43.918030 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:43.920907 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:43.924709 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:43.924721 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:43.924773 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:43.925280 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:43.925732 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:43.925738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:43.925742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:43.925960 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:43.926562 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:43.926572 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:43.926963 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 19:12:43.927262 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 19:12:43.927860 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":310,"threads_started":1}
I20260924 19:12:43.930891 15144 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38233
I20260924 19:12:43.931254 15144 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:12:43.932138 15144 heartbeater.cc:511] Master 127.11.18.126:38233 requested a full tablet report, sending...
I20260924 19:12:43.934594 14955 ts_manager.cc:194] Registered new tserver with Master: 6132a855bed44e27bd2d690e4f5a83f0 (127.11.18.65:45475)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
I20260924 19:12:43.936311 14955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:41121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:12:43.964933 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38233
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:44.321785 15150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:44.322124 15150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:44.322295 15150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:44.330564 15150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:44.330691 15150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:44.330874 15150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:44.341115 15150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:44.342518 15150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:44.343873 15150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:44.357472 15155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:44.357451 15156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:44.357568 15158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:44.358945 15150 server_base.cc:1034] running on GCE node
I20260924 19:12:44.359728 15150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:44.360859 15150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:44.362113 15150 hybrid_clock.cc:648] HybridClock initialized: now 1790277164362031 us; error 64 us; skew 500 ppm
I20260924 19:12:44.362519 15150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:44 dist-test-slave-0lgr krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277164, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:44.366473 15150 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:12:44.368404 15150 webserver.cc:460] Webserver started at http://127.11.18.66:37085/ using document root <none> and password file <none>
I20260924 19:12:44.369179 15150 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:44.369311 15150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:44.369681 15150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:44.372771 15150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "140d8fd2fda14a7f931e0056fc33a04c"
format_stamp: "Formatted at 2026-09-24 19:12:44 on dist-test-slave-0lgr"
server_key: "6d1806d7725e6af2eede53bb212af4c2"
server_key_iv: "76334557a39a39bbc5c10051f848bc81"
server_key_version: "encryptionkey@0"
I20260924 19:12:44.373667 15150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "140d8fd2fda14a7f931e0056fc33a04c"
format_stamp: "Formatted at 2026-09-24 19:12:44 on dist-test-slave-0lgr"
server_key: "6d1806d7725e6af2eede53bb212af4c2"
server_key_iv: "76334557a39a39bbc5c10051f848bc81"
server_key_version: "encryptionkey@0"
I20260924 19:12:44.379482 15150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:12:44.383598 15165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:44.385445 15150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 19:12:44.385653 15150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "140d8fd2fda14a7f931e0056fc33a04c"
format_stamp: "Formatted at 2026-09-24 19:12:44 on dist-test-slave-0lgr"
server_key: "6d1806d7725e6af2eede53bb212af4c2"
server_key_iv: "76334557a39a39bbc5c10051f848bc81"
server_key_version: "encryptionkey@0"
I20260924 19:12:44.385864 15150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:44.411685 15150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:44.416778 15150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:44.417133 15150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:44.418352 15150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:44.419896 15150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:44.419977 15150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:44.420115 15150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:44.420202 15150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:44.458453 15150 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:42557
I20260924 19:12:44.458549 15277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:42557 every 8 connection(s)
I20260924 19:12:44.459846 15150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:44.468568 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 15150
I20260924 19:12:44.468777 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:44.469336 11337 external_mini_cluster.cc:1428] Setting key 47322cfd587440d8c4f479910b00dee8
Sep 24 19:12:44 dist-test-slave-0lgr krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277164, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:44.482618 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:44.462803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:43503 (local address 127.11.18.126:38233)
0924 19:12:44.463265 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:44.463276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:44.464355 (+ 1079us) server_negotiation.cc:407] Connection header received
0924 19:12:44.466324 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:44.466329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:44.466403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:44.466612 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:44.468243 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:44.469223 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:44.470971 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:44.471542 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:44.471717 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:44.475990 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:44.475995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:44.475999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:44.476033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:44.479119 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:44.479955 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:44.479961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:44.479971 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:44.480103 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:44.480586 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:44.480592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:44.480602 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:44.480902 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:44.481074 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:44.481940 (+ 866us) server_negotiation.cc:299] Negotiation successful
0924 19:12:44.482191 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260924 19:12:44.483790 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:44.463449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38233 (local address 127.11.18.66:43503)
0924 19:12:44.464183 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:44.464221 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:44.466043 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:44.466884 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:44.466893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:44.467419 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:44.468036 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:44.468060 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:44.469429 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:44.469437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:44.470828 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:44.470841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:44.471782 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:44.471790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:44.472114 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:44.472872 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:44.472908 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:44.475692 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:44.479304 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:44.479314 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:44.479335 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:44.479807 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:44.480225 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:44.480232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:44.480236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:44.480451 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:44.481048 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:44.481058 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:44.481521 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 19:12:44.481825 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 19:12:44.482431 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":299,"threads_started":1}
I20260924 19:12:44.485740 15278 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38233
I20260924 19:12:44.486147 15278 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:44.486959 15278 heartbeater.cc:511] Master 127.11.18.126:38233 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:12:44.488673 14955 ts_manager.cc:194] Registered new tserver with Master: 140d8fd2fda14a7f931e0056fc33a04c (127.11.18.66:42557)
I20260924 19:12:44.489792 14955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:43503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:12:44.517884 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38233
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:44.862129 15284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:44.862412 15284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:44.862504 15284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:44.870148 15284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:44.870265 15284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:44.870380 15284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:44.880556 15284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:44.881834 15284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:44.883155 15284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:44.897360 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:44.897400 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:44.897439 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:44.899034 15284 server_base.cc:1034] running on GCE node
I20260924 19:12:44.899834 15284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:44.901054 15284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:44.902365 15284 hybrid_clock.cc:648] HybridClock initialized: now 1790277164902283 us; error 87 us; skew 500 ppm
I20260924 19:12:44.902770 15284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:44 dist-test-slave-0lgr krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277164, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:44.906662 15284 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:12:44.909629 15284 webserver.cc:460] Webserver started at http://127.11.18.67:45927/ using document root <none> and password file <none>
I20260924 19:12:44.910329 15284 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:44.910480 15284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:44.910799 15284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:44.913872 15284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "b4db9ae6971241ee8ec68ce2741c7a4a"
format_stamp: "Formatted at 2026-09-24 19:12:44 on dist-test-slave-0lgr"
server_key: "3fef9f4d3aec250924388c66d7d23ebc"
server_key_iv: "1631e9180fef7dad080006c0b94bc581"
server_key_version: "encryptionkey@0"
I20260924 19:12:44.914669 15284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "b4db9ae6971241ee8ec68ce2741c7a4a"
format_stamp: "Formatted at 2026-09-24 19:12:44 on dist-test-slave-0lgr"
server_key: "3fef9f4d3aec250924388c66d7d23ebc"
server_key_iv: "1631e9180fef7dad080006c0b94bc581"
server_key_version: "encryptionkey@0"
I20260924 19:12:44.920360 15284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:12:44.924124 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:44.925766 15284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:44.925947 15284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "b4db9ae6971241ee8ec68ce2741c7a4a"
format_stamp: "Formatted at 2026-09-24 19:12:44 on dist-test-slave-0lgr"
server_key: "3fef9f4d3aec250924388c66d7d23ebc"
server_key_iv: "1631e9180fef7dad080006c0b94bc581"
server_key_version: "encryptionkey@0"
I20260924 19:12:44.926162 15284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:44.939713 15144 heartbeater.cc:503] Master 127.11.18.126:38233 was elected leader, sending a full tablet report...
I20260924 19:12:44.949154 15284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:44.953954 15284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:44.954316 15284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:44.955353 15284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:44.956704 15284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:44.956816 15284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:44.956959 15284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:44.957032 15284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:44.996438 15284 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:38491
I20260924 19:12:44.996493 15411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:38491 every 8 connection(s)
I20260924 19:12:44.997838 15284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:44.999188 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 15284
I20260924 19:12:44.999416 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:44.999814 11337 external_mini_cluster.cc:1428] Setting key 15c5b56710c60f230e12a64cfdf81496
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277164, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:45.021210 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:45.002086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:53013 (local address 127.11.18.126:38233)
0924 19:12:45.002596 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:45.002602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:45.003924 (+ 1322us) server_negotiation.cc:407] Connection header received
0924 19:12:45.005735 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:45.005739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:45.005803 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:45.005910 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:45.007407 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.008200 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.009899 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.010336 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.010448 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:45.014795 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:45.014804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:45.014809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:45.014858 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:45.017907 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.018727 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.018737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.018739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.018858 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.019386 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.019390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.019399 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.019664 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:45.019836 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:45.020604 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 19:12:45.020843 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
I20260924 19:12:45.022238 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:45.002772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38233 (local address 127.11.18.67:53013)
0924 19:12:45.003752 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:45.003814 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:45.005488 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:45.006104 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:45.006113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:45.006614 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:45.007239 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:45.007257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.008360 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.008364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:45.009701 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:45.009713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.010548 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.010553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:45.010753 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:45.011563 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:45.011587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:45.014544 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:45.018063 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:45.018074 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:45.018094 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:45.018552 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:45.019008 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:45.019014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:45.019017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:45.019263 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:45.019811 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:45.019819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:45.020200 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 19:12:45.020517 (+ 317us) client_negotiation.cc:240] Negotiation successful
0924 19:12:45.021079 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":374,"threads_started":1}
I20260924 19:12:45.023927 15412 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38233
I20260924 19:12:45.024288 15412 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:45.025130 15412 heartbeater.cc:511] Master 127.11.18.126:38233 requested a full tablet report, sending...
I20260924 19:12:45.026834 14955 ts_manager.cc:194] Registered new tserver with Master: b4db9ae6971241ee8ec68ce2741c7a4a (127.11.18.67:38491)
I20260924 19:12:45.027822 14955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:53013
I20260924 19:12:45.028843 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:45.080636 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:45.063054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.11.18.65:45475)
0924 19:12:45.063853 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:45.063864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:45.063897 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:12:45.064151 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:45.064159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:45.064387 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:45.064595 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:45.066687 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.067726 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.069060 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.069757 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.069903 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:45.073905 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:45.073919 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:45.073934 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:45.073975 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:45.077006 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.078094 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.078104 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.078106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.078258 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.078737 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.078748 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.078750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.079213 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:45.079590 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:45.079907 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 19:12:45.080198 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":317,"threads_started":1}
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:45.101156 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:45.086023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.11.18.126:38233)
0924 19:12:45.086242 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:45.086247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:45.086503 (+ 256us) server_negotiation.cc:407] Connection header received
0924 19:12:45.086598 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:45.086602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:45.086743 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:45.086826 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:45.088185 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.089013 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.090715 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.091246 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.091439 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:45.095589 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:45.095595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:45.095598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:45.095637 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:45.098436 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.099175 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.099183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.099187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.099320 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.099704 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.099710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.099713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.100049 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:45.100289 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:45.100598 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 19:12:45.100825 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 19:12:45.101781 14954 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47550
I20260924 19:12:45.102950 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 15016
I20260924 19:12:45.117476 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:45.104028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.11.18.65:45475)
0924 19:12:45.104212 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:45.104216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:45.104269 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:12:45.104404 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:45.104408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:45.104481 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:45.104564 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:45.105971 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.106667 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.108065 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.108553 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.108739 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:45.111241 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:45.111246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:45.111249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:45.111285 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:45.114602 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.115320 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.115325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.115327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.115467 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.116018 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.116022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.116024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.116292 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:45.116478 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:45.116860 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 19:12:45.117061 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:12:45.118160 15139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:45.206797 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 15016
I20260924 19:12:45.223104 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 15150
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:12:45.240890 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:45.224077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.11.18.66:42557)
0924 19:12:45.224694 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:45.224704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:45.224735 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:12:45.224811 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:45.224819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:45.225095 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:45.225252 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:45.226456 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.228060 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.229553 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.230252 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.230382 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:45.234809 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:45.234822 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:45.234841 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:45.234888 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:45.238173 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.238896 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.238904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.238906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.239028 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.239395 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.239398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.239400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.239790 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:45.240010 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:45.240391 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 19:12:45.240596 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":316,"threads_started":1}
I20260924 19:12:45.242262 15273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:45.327517 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 15150
I20260924 19:12:45.343971 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 15284
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[14910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:12:45.365473 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:45.345100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.11.18.67:38491)
0924 19:12:45.345755 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:45.345762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:45.345848 (+ 86us) server_negotiation.cc:407] Connection header received
0924 19:12:45.345908 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:45.345916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:45.346126 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:45.346248 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:45.347603 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.348664 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.350176 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:45.350748 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:45.350912 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:45.359339 (+ 8427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:45.359355 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:45.359371 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:45.359415 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:45.362419 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.363277 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.363284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.363286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.363417 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:45.363793 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:45.363795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:45.363802 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:45.364147 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:45.364374 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:45.364700 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 19:12:45.364952 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":372,"threads_started":1}
I20260924 19:12:45.366942 15407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:45.450412 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 15284
I20260924 19:12:45.466025 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 14923
I20260924 19:12:45.467298 14985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:45.574357 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 14923
2026-09-24T19:12:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4209 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[15432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[15432](info): set up 2 sockets
Sep 24 19:12:45 dist-test-slave-0lgr krb5kdc[15432](info): commencing operation
krb5kdc: starting...
W20260924 19:12:46.770550 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:46 dist-test-slave-0lgr krb5kdc[15432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:46.994681 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:37507
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:47.332141 15445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:47.332518 15445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:47.332683 15445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:47.340044 15445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:47.340221 15445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:47.340328 15445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:47.340448 15445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:47.340536 15445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:47.350884 15445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:47.352726 15445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:47.354708 15445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:47.368048 15450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:47.368072 15451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:47.368307 15453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:47.369495 15445 server_base.cc:1034] running on GCE node
I20260924 19:12:47.370412 15445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:47.372249 15445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:47.373512 15445 hybrid_clock.cc:648] HybridClock initialized: now 1790277167373453 us; error 50 us; skew 500 ppm
I20260924 19:12:47.373963 15445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:47 dist-test-slave-0lgr krb5kdc[15432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277167, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:47.377650 15445 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:47.379163 15445 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:47.379263 15445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:47.379621 15445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:47.383111 15445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "9355ec1550fd4b6aa356c6faff7d6253"
format_stamp: "Formatted at 2026-09-24 19:12:47 on dist-test-slave-0lgr"
server_key: "57e3fc63bc347f16be26608c11a5af4b"
server_key_iv: "35b89cd34900097db8323e7b9eeef094"
server_key_version: "encryptionkey@0"
I20260924 19:12:47.384045 15445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "9355ec1550fd4b6aa356c6faff7d6253"
format_stamp: "Formatted at 2026-09-24 19:12:47 on dist-test-slave-0lgr"
server_key: "57e3fc63bc347f16be26608c11a5af4b"
server_key_iv: "35b89cd34900097db8323e7b9eeef094"
server_key_version: "encryptionkey@0"
I20260924 19:12:47.389868 15445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:12:47.393795 15459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:47.395294 15445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:47.395493 15445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "9355ec1550fd4b6aa356c6faff7d6253"
format_stamp: "Formatted at 2026-09-24 19:12:47 on dist-test-slave-0lgr"
server_key: "57e3fc63bc347f16be26608c11a5af4b"
server_key_iv: "35b89cd34900097db8323e7b9eeef094"
server_key_version: "encryptionkey@0"
I20260924 19:12:47.395807 15445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:47.408457 15445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:47.413697 15445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:47.414026 15445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:47.431690 15445 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:37507
I20260924 19:12:47.431711 15510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:37507 every 8 connection(s)
I20260924 19:12:47.432602 15445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:47.434971 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 15445
I20260924 19:12:47.435227 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:47.435923 11337 external_mini_cluster.cc:1428] Setting key 7dc9d649961e553c940c4aa63b8f8561
I20260924 19:12:47.437208 15511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:12:47 dist-test-slave-0lgr krb5kdc[15432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:47.451258 15511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253: Bootstrap starting.
I20260924 19:12:47.455240 15511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:47.456622 15511 log.cc:824] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:47.460249 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:47.437891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.11.18.126:37507)
0924 19:12:47.438710 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:47.438728 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:47.439155 (+ 427us) server_negotiation.cc:407] Connection header received
0924 19:12:47.441023 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:47.441045 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:47.441515 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:47.441941 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:47.443424 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:47.444925 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:47.446280 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:47.446948 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:47.447152 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:47.451801 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:47.451810 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:47.451825 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:47.451868 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:47.455052 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:47.456045 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:47.456052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:47.456058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:47.456297 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:47.456864 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:47.456871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:47.456875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:47.457566 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:47.457812 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:47.458460 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 19:12:47.458913 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":354,"threads_started":1}
I20260924 19:12:47.460362 15511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253: No bootstrap required, opened a new log
I20260924 19:12:47.468839 15511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9355ec1550fd4b6aa356c6faff7d6253" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37507 } }
I20260924 19:12:47.469197 15511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:47.469264 15511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9355ec1550fd4b6aa356c6faff7d6253, State: Initialized, Role: FOLLOWER
I20260924 19:12:47.469887 15511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9355ec1550fd4b6aa356c6faff7d6253" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37507 } }
I20260924 19:12:47.470090 15511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:47.470180 15511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:47.470307 15511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:47.472404 15511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9355ec1550fd4b6aa356c6faff7d6253" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37507 } }
I20260924 19:12:47.472862 15511 leader_election.cc:302] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9355ec1550fd4b6aa356c6faff7d6253; no voters:
I20260924 19:12:47.473531 15511 leader_election.cc:288] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:47.473642 15516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:47.474421 15516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [term 1 LEADER]: Becoming Leader. State: Replica: 9355ec1550fd4b6aa356c6faff7d6253, State: Running, Role: LEADER
I20260924 19:12:47.474968 15516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9355ec1550fd4b6aa356c6faff7d6253" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37507 } }
I20260924 19:12:47.475723 15511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:47.478186 15518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9355ec1550fd4b6aa356c6faff7d6253. Latest consensus state: current_term: 1 leader_uuid: "9355ec1550fd4b6aa356c6faff7d6253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9355ec1550fd4b6aa356c6faff7d6253" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37507 } } }
I20260924 19:12:47.478675 15518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:47.479225 15517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9355ec1550fd4b6aa356c6faff7d6253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9355ec1550fd4b6aa356c6faff7d6253" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 37507 } } }
I20260924 19:12:47.479604 15517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:47.480584 15524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:47.486189 15524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:47.495595 15524 catalog_manager.cc:1347] Generated new cluster ID: 6bb333b2e5ef4d1f8761f01a12be5892
I20260924 19:12:47.495765 15524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:47.507241 15524 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:47.509253 15524 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:47.518671 15524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253: Generated new TSK 0
I20260924 19:12:47.519646 15524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:12:47.586576 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37507
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:47.943039 15537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:47.943408 15537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:47.943584 15537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:47.951213 15537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:47.951385 15537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:47.951577 15537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:47.961907 15537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:47.963281 15537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:47.964567 15537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:47.979400 15543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:47.979444 15545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:47.979367 15542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:47.983167 15537 server_base.cc:1034] running on GCE node
I20260924 19:12:47.983963 15537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:47.985052 15537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:47.986269 15537 hybrid_clock.cc:648] HybridClock initialized: now 1790277167986196 us; error 67 us; skew 500 ppm
I20260924 19:12:47.986670 15537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:47 dist-test-slave-0lgr krb5kdc[15432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277167, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:47.990681 15537 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:12:47.992319 15537 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:47.992436 15537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:47.992731 15537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:47.996150 15537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "edc78a35bff341cb936994588b93bf2f"
format_stamp: "Formatted at 2026-09-24 19:12:47 on dist-test-slave-0lgr"
server_key: "f81ae9dd75deec54a62e8c12da42c8a4"
server_key_iv: "ec84ae4d8718137eb3ce8ece36403d63"
server_key_version: "encryptionkey@0"
I20260924 19:12:47.997040 15537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "edc78a35bff341cb936994588b93bf2f"
format_stamp: "Formatted at 2026-09-24 19:12:47 on dist-test-slave-0lgr"
server_key: "f81ae9dd75deec54a62e8c12da42c8a4"
server_key_iv: "ec84ae4d8718137eb3ce8ece36403d63"
server_key_version: "encryptionkey@0"
I20260924 19:12:48.003597 15537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:12:48.007560 15551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:48.009197 15537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:12:48.009464 15537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "edc78a35bff341cb936994588b93bf2f"
format_stamp: "Formatted at 2026-09-24 19:12:47 on dist-test-slave-0lgr"
server_key: "f81ae9dd75deec54a62e8c12da42c8a4"
server_key_iv: "ec84ae4d8718137eb3ce8ece36403d63"
server_key_version: "encryptionkey@0"
I20260924 19:12:48.009819 15537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:48.034899 15537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:48.039505 15537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:48.039821 15537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:48.040920 15537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:48.042517 15537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:48.042644 15537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:48.042788 15537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:48.042878 15537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:48.080197 15537 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:41173
I20260924 19:12:48.080302 15663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:41173 every 8 connection(s)
I20260924 19:12:48.081282 15537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:48.090230 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 15537
I20260924 19:12:48.090490 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:48.091207 11337 external_mini_cluster.cc:1428] Setting key d230c3f75ff4c67e8c04a638f068e28e
Sep 24 19:12:48 dist-test-slave-0lgr krb5kdc[15432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277167, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:48.104635 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:48.084448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:39957 (local address 127.11.18.126:37507)
0924 19:12:48.084911 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:48.084916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:48.086056 (+ 1140us) server_negotiation.cc:407] Connection header received
0924 19:12:48.088108 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:48.088114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:48.088206 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:48.088330 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:48.089959 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:48.090797 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:48.092598 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:48.093018 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:48.093170 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:48.097079 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:48.097090 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:48.097099 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:48.097161 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:48.100851 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:48.101947 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:48.101951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:48.101954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:48.102081 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:48.102695 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:48.102700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:48.102702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:48.103024 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:48.103156 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:48.104127 (+ 971us) server_negotiation.cc:299] Negotiation successful
0924 19:12:48.104316 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260924 19:12:48.106099 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:48.085070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37507 (local address 127.11.18.65:39957)
0924 19:12:48.085881 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:48.085925 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:48.087774 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:48.088569 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:48.088586 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:48.089118 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:48.089781 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:48.089798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:48.090957 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:48.090967 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:48.092443 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:48.092455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:48.093257 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:48.093264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:48.093530 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:48.094246 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:48.094269 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:48.096813 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:48.101023 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:48.101036 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:48.101058 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:48.101782 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:48.102238 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:48.102245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:48.102252 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:48.102503 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:48.103165 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:48.103176 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:48.103647 (+ 471us) client_negotiation.cc:769] Sending connection context
0924 19:12:48.103970 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 19:12:48.104581 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":351,"threads_started":1}
I20260924 19:12:48.107862 15664 heartbeater.cc:347] Connected to a master server at 127.11.18.126:37507
I20260924 19:12:48.108240 15664 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:48.108986 15664 heartbeater.cc:511] Master 127.11.18.126:37507 requested a full tablet report, sending...
I20260924 19:12:48.111420 15476 ts_manager.cc:194] Registered new tserver with Master: edc78a35bff341cb936994588b93bf2f (127.11.18.65:41173)
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:12:48.113055 15476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:39957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:12:48.143786 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37507
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:48.500852 15670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:48.501209 15670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:48.501407 15670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:48.508906 15670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:48.509084 15670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:48.509289 15670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:48.519668 15670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:48.521045 15670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:48.522498 15670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:48.536387 15676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:48.536361 15675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:48.536339 15678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:48.537113 15670 server_base.cc:1034] running on GCE node
I20260924 19:12:48.537853 15670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:48.538928 15670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:48.540141 15670 hybrid_clock.cc:648] HybridClock initialized: now 1790277168540081 us; error 84 us; skew 500 ppm
I20260924 19:12:48.540524 15670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:48 dist-test-slave-0lgr krb5kdc[15432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277168, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:48.549386 15670 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:12:48.550824 15670 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:48.550985 15670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:48.551342 15670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:48.554740 15670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "a2d459686fc74baeaeb8d5b17d5482bc"
format_stamp: "Formatted at 2026-09-24 19:12:48 on dist-test-slave-0lgr"
server_key: "d70b178f111b11daf0fa2101c05ef841"
server_key_iv: "579cfc17a84f99721a842c6be236f1ba"
server_key_version: "encryptionkey@0"
I20260924 19:12:48.555672 15670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "a2d459686fc74baeaeb8d5b17d5482bc"
format_stamp: "Formatted at 2026-09-24 19:12:48 on dist-test-slave-0lgr"
server_key: "d70b178f111b11daf0fa2101c05ef841"
server_key_iv: "579cfc17a84f99721a842c6be236f1ba"
server_key_version: "encryptionkey@0"
I20260924 19:12:48.561861 15670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:12:48.566172 15684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:48.567862 15670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:12:48.568086 15670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "a2d459686fc74baeaeb8d5b17d5482bc"
format_stamp: "Formatted at 2026-09-24 19:12:48 on dist-test-slave-0lgr"
server_key: "d70b178f111b11daf0fa2101c05ef841"
server_key_iv: "579cfc17a84f99721a842c6be236f1ba"
server_key_version: "encryptionkey@0"
I20260924 19:12:48.568353 15670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:48.590080 15670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:48.594601 15670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:48.594930 15670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:48.595978 15670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:48.597464 15670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:48.597595 15670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:48.597741 15670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:48.597833 15670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:48.632951 15670 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:46411
I20260924 19:12:48.633068 15796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:46411 every 8 connection(s)
I20260924 19:12:48.633888 15670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:48.635876 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 15670
I20260924 19:12:48.636070 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:48.636554 11337 external_mini_cluster.cc:1428] Setting key fd213da53b313bf0dad00b2bea74d26b
Sep 24 19:12:48 dist-test-slave-0lgr krb5kdc[15432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277168, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:12:48.660027 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:48.637832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:58835 (local address 127.11.18.126:37507)
0924 19:12:48.638299 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:48.638305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:48.641424 (+ 3119us) server_negotiation.cc:407] Connection header received
0924 19:12:48.641498 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:48.641506 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:48.641597 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:48.641715 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:48.643665 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:48.644866 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:48.646571 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:48.647289 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:48.647486 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:48.652007 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:48.652018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:48.652027 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:48.652081 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:48.656229 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:48.657140 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:48.657148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:48.657155 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:48.657352 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:48.657975 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:48.657982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:48.657988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:48.658378 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:48.658574 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:48.659486 (+ 912us) server_negotiation.cc:299] Negotiation successful
0924 19:12:48.659753 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":998,"threads_started":1}
I20260924 19:12:48.661053 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:48.637367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37507 (local address 127.11.18.66:58835)
0924 19:12:48.638335 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:48.638418 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:48.641002 (+ 2584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:48.641990 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:48.642000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:48.642617 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:48.643420 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:48.643448 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:48.645038 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:48.645045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:48.646393 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:48.646406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:48.647631 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:48.647641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:48.647932 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:48.648742 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:48.648768 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:48.651655 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:48.656426 (+ 4771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:48.656436 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:48.656457 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:48.656971 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:48.657534 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:48.657537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:48.657540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:48.657815 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:48.658674 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:48.658681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:48.659028 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 19:12:48.659323 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 19:12:48.659829 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"spinlock_wait_cycles":22912,"thread_start_us":516,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:12:48.662976 15797 heartbeater.cc:347] Connected to a master server at 127.11.18.126:37507
I20260924 19:12:48.663321 15797 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:48.664100 15797 heartbeater.cc:511] Master 127.11.18.126:37507 requested a full tablet report, sending...
I20260924 19:12:48.665750 15476 ts_manager.cc:194] Registered new tserver with Master: a2d459686fc74baeaeb8d5b17d5482bc (127.11.18.66:46411)
I20260924 19:12:48.667122 15476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:58835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:12:48.689576 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37507
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:49.021232 15803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:49.021616 15803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:49.021801 15803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:49.029232 15803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:49.029402 15803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:49.029619 15803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:49.040099 15803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:37507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:49.041447 15803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:49.042743 15803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:49.056162 15808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:49.056352 15809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:49.056389 15811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:49.057178 15803 server_base.cc:1034] running on GCE node
I20260924 19:12:49.057955 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:49.058984 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:49.060240 15803 hybrid_clock.cc:648] HybridClock initialized: now 1790277169060217 us; error 58 us; skew 500 ppm
I20260924 19:12:49.060595 15803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:49 dist-test-slave-0lgr krb5kdc[15432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277169, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:49.063988 15803 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:12:49.065243 15803 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:49.065403 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:49.065794 15803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:49.068730 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "d45fafcc6776405897ca07a819269d98"
format_stamp: "Formatted at 2026-09-24 19:12:49 on dist-test-slave-0lgr"
server_key: "c74e419562c17a9b913eeb37f5b1c231"
server_key_iv: "462e7220b91954fd3d10550d3dc3a812"
server_key_version: "encryptionkey@0"
I20260924 19:12:49.069629 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "d45fafcc6776405897ca07a819269d98"
format_stamp: "Formatted at 2026-09-24 19:12:49 on dist-test-slave-0lgr"
server_key: "c74e419562c17a9b913eeb37f5b1c231"
server_key_iv: "462e7220b91954fd3d10550d3dc3a812"
server_key_version: "encryptionkey@0"
I20260924 19:12:49.074909 15803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:12:49.078663 15817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:49.080380 15803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:49.080587 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "d45fafcc6776405897ca07a819269d98"
format_stamp: "Formatted at 2026-09-24 19:12:49 on dist-test-slave-0lgr"
server_key: "c74e419562c17a9b913eeb37f5b1c231"
server_key_iv: "462e7220b91954fd3d10550d3dc3a812"
server_key_version: "encryptionkey@0"
I20260924 19:12:49.080792 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:49.100531 15803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:49.105012 15803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:49.105305 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:49.106383 15803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:49.107782 15803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:49.107872 15803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:49.108026 15803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:49.108108 15803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:49.117225 15664 heartbeater.cc:503] Master 127.11.18.126:37507 was elected leader, sending a full tablet report...
I20260924 19:12:49.145402 15803 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:42345
I20260924 19:12:49.145529 15929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:42345 every 8 connection(s)
I20260924 19:12:49.146308 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:49.150322 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 15803
I20260924 19:12:49.150643 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:49.151144 11337 external_mini_cluster.cc:1428] Setting key ed646bbf48eb50b1bb14c11ddf9be81b
Sep 24 19:12:49 dist-test-slave-0lgr krb5kdc[15432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277169, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:49.167955 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.149227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:36947 (local address 127.11.18.126:37507)
0924 19:12:49.149521 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.149527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.150759 (+ 1232us) server_negotiation.cc:407] Connection header received
0924 19:12:49.152788 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.152794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.152889 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.153015 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:49.155044 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.155892 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.157443 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.157863 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.157970 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.161761 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:49.161766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:49.161770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:49.161808 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:49.164440 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.165406 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.165411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.165415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.165563 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.166086 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.166090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.166092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.166465 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:49.166582 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.167456 (+ 874us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.167620 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260924 19:12:49.168854 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.149740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:37507 (local address 127.11.18.67:36947)
0924 19:12:49.150585 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:49.150636 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:49.152526 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:49.153280 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:49.153290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:49.154166 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:49.154863 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.154880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.156036 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.156040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.157255 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.157266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.158101 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.158106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.158303 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.159025 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:49.159046 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:49.161516 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:49.164633 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:49.164642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:49.164659 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:49.165222 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:49.165712 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:49.165718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:49.165722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:49.165960 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:49.166611 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:49.166620 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:49.167011 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 19:12:49.167368 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 19:12:49.167785 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":416,"threads_started":1}
I20260924 19:12:49.170430 15930 heartbeater.cc:347] Connected to a master server at 127.11.18.126:37507
I20260924 19:12:49.170753 15930 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:49.171432 15930 heartbeater.cc:511] Master 127.11.18.126:37507 requested a full tablet report, sending...
I20260924 19:12:49.172894 15476 ts_manager.cc:194] Registered new tserver with Master: d45fafcc6776405897ca07a819269d98 (127.11.18.67:42345)
I20260924 19:12:49.174003 15476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:36947
I20260924 19:12:49.180106 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:12:49.199756 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.185140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.11.18.126:37507)
0924 19:12:49.185445 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.185451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.185940 (+ 489us) server_negotiation.cc:407] Connection header received
0924 19:12:49.186016 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.186020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.186095 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.186204 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:49.187767 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.188596 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.190216 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.190670 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.190793 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.193236 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:49.193241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:49.193245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:49.193286 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:49.196271 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.197039 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.197043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.197046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.197215 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.197832 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.197837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.197839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.198190 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:49.198663 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.199113 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.199327 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260924 19:12:49.205094 15476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:12:49.212780 15476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:49.252138 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.226966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.11.18.66:46411)
0924 19:12:49.242099 (+ 15133us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.242107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.242132 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:12:49.242223 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.242228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.242419 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.242655 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:49.243607 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.244724 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.248114 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.250279 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.250405 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.250664 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.251428 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.251632 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14988,"thread_start_us":15912,"threads_started":1}
I20260924 19:12:49.253060 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.226724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:46411 (local address 127.0.0.1:47358)
0924 19:12:49.239852 (+ 13128us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:49.239928 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:49.240222 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:49.242685 (+ 2463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:49.242694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:49.242737 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:49.243365 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.243389 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.244889 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.244896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.247893 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.247910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.250891 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.250899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.251126 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.251185 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 19:12:49.252423 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0924 19:12:49.252670 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12707,"threads_started":1}
I20260924 19:12:49.256125 15946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.227034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:41173 (local address 127.0.0.1:33632)
0924 19:12:49.245867 (+ 18833us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:49.245889 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:49.246026 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:49.247993 (+ 1967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:49.248000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:49.248032 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:49.248494 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.248507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.250582 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.250588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.252963 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.252978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.255228 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.255235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.255441 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.255482 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:12:49.255610 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 19:12:49.255762 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18688,"thread_start_us":13477,"threads_started":1}
I20260924 19:12:49.256531 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.227316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.11.18.65:41173)
0924 19:12:49.246619 (+ 19303us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.247078 (+ 459us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.247118 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:12:49.247233 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.247242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.247519 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.247721 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:49.248694 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.250416 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.253368 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.255001 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.255165 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.255414 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.255867 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.256047 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19116,"thread_start_us":18957,"threads_started":1}
I20260924 19:12:49.257834 15732 tablet_service.cc:1467] Processing CreateTablet for tablet 22252e0c803e4ef9b530c9e5c9d8a299 (DEFAULT_TABLE table=test-table [id=d4f6c189188c46a69946ccfce4a69103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:49.259449 15732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22252e0c803e4ef9b530c9e5c9d8a299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:49.260330 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 22252e0c803e4ef9b530c9e5c9d8a299 (DEFAULT_TABLE table=test-table [id=d4f6c189188c46a69946ccfce4a69103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:49.261833 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22252e0c803e4ef9b530c9e5c9d8a299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:49.263029 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.227369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:42345 (local address 127.0.0.1:35680)
0924 19:12:49.253349 (+ 25980us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:49.253377 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:49.253504 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:49.254151 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:49.254157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:49.254187 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:49.254348 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.254358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.258578 (+ 4220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.258586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.260554 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.260567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.262136 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.262143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.262350 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.262387 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:12:49.262519 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 19:12:49.262708 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25810,"thread_start_us":18184,"threads_started":1}
I20260924 19:12:49.263108 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.227929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.11.18.67:42345)
0924 19:12:49.248765 (+ 20836us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.248779 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.253540 (+ 4761us) server_negotiation.cc:407] Connection header received
0924 19:12:49.253665 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.253672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.253919 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.254158 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:49.254551 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.258399 (+ 3848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.260748 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.261943 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.262118 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.262376 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.262560 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.262746 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20580,"thread_start_us":20852,"threads_started":1}
I20260924 19:12:49.267030 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 22252e0c803e4ef9b530c9e5c9d8a299 (DEFAULT_TABLE table=test-table [id=d4f6c189188c46a69946ccfce4a69103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:49.268261 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22252e0c803e4ef9b530c9e5c9d8a299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:49.275990 15948 tablet_bootstrap.cc:492] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Bootstrap starting.
I20260924 19:12:49.278241 15949 tablet_bootstrap.cc:492] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Bootstrap starting.
I20260924 19:12:49.281675 15948 tablet_bootstrap.cc:654] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:49.281703 15949 tablet_bootstrap.cc:654] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:49.282178 15950 tablet_bootstrap.cc:492] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Bootstrap starting.
I20260924 19:12:49.283079 15949 log.cc:824] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:49.283524 15948 log.cc:824] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:49.286199 15950 tablet_bootstrap.cc:654] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:49.286952 15948 tablet_bootstrap.cc:492] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: No bootstrap required, opened a new log
I20260924 19:12:49.287250 15948 ts_tablet_manager.cc:1397] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260924 19:12:49.287473 15950 log.cc:824] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:49.290534 15950 tablet_bootstrap.cc:492] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: No bootstrap required, opened a new log
I20260924 19:12:49.290818 15950 ts_tablet_manager.cc:1397] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 19:12:49.295288 15948 raft_consensus.cc:348] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.295693 15948 raft_consensus.cc:374] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:49.295809 15948 raft_consensus.cc:729] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d459686fc74baeaeb8d5b17d5482bc, State: Initialized, Role: FOLLOWER
I20260924 19:12:49.296497 15948 consensus_queue.cc:260] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.298158 15949 tablet_bootstrap.cc:492] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: No bootstrap required, opened a new log
I20260924 19:12:49.298358 15948 ts_tablet_manager.cc:1428] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260924 19:12:49.298460 15949 ts_tablet_manager.cc:1397] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.007s
I20260924 19:12:49.298856 15797 heartbeater.cc:503] Master 127.11.18.126:37507 was elected leader, sending a full tablet report...
I20260924 19:12:49.299686 15950 raft_consensus.cc:348] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.300141 15950 raft_consensus.cc:374] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:49.300323 15950 raft_consensus.cc:729] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45fafcc6776405897ca07a819269d98, State: Initialized, Role: FOLLOWER
I20260924 19:12:49.300984 15950 consensus_queue.cc:260] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.302201 15930 heartbeater.cc:503] Master 127.11.18.126:37507 was elected leader, sending a full tablet report...
I20260924 19:12:49.302834 15950 ts_tablet_manager.cc:1428] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 19:12:49.309731 15949 raft_consensus.cc:348] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.310105 15949 raft_consensus.cc:374] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:49.310241 15949 raft_consensus.cc:729] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc78a35bff341cb936994588b93bf2f, State: Initialized, Role: FOLLOWER
I20260924 19:12:49.310941 15949 consensus_queue.cc:260] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.313148 15949 ts_tablet_manager.cc:1428] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260924 19:12:49.333806 15665 tablet.cc:2369] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:12:49.350888 15956 raft_consensus.cc:482] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:49.351235 15956 raft_consensus.cc:504] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.352928 15956 leader_election.cc:288] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2d459686fc74baeaeb8d5b17d5482bc (127.11.18.66:46411), d45fafcc6776405897ca07a819269d98 (127.11.18.67:42345)
I20260924 19:12:49.359884 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.353528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:40579 (local address 127.11.18.67:42345)
0924 19:12:49.353771 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.353794 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.354537 (+ 743us) server_negotiation.cc:407] Connection header received
0924 19:12:49.354738 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.354741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.354799 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.354897 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:49.355587 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.356284 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.358167 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.358947 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.359141 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.359215 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.359482 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.359613 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 19:12:49.360133 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.353421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:42345 (local address 127.11.18.65:40579)
0924 19:12:49.354389 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:49.354404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:49.354587 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:49.354924 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:49.354929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:49.354953 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:49.355412 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.355423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.356425 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.356432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.358010 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.358022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.359152 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.359157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.359316 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.359339 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 19:12:49.359456 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 19:12:49.359655 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":80,"thread_start_us":595,"threads_started":1}
I20260924 19:12:49.360237 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.353348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:40063 (local address 127.11.18.66:46411)
0924 19:12:49.353578 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.353584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.354408 (+ 824us) server_negotiation.cc:407] Connection header received
0924 19:12:49.354646 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.354657 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.354735 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.354851 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:12:49.355524 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.356338 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.358367 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.359201 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.359399 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.359479 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.359811 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.359944 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:12:49.360543 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.353298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:46411 (local address 127.11.18.65:40063)
0924 19:12:49.354276 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:49.354294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:49.354494 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:49.354868 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:49.354873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:49.354901 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:12:49.355365 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.355375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.356494 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.356499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.358193 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:49.358206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.359445 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.359452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:49.359634 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.359659 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:12:49.359787 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 19:12:49.359925 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":29,"thread_start_us":586,"threads_started":1}
I20260924 19:12:49.361487 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22252e0c803e4ef9b530c9e5c9d8a299" candidate_uuid: "edc78a35bff341cb936994588b93bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45fafcc6776405897ca07a819269d98" is_pre_election: true
I20260924 19:12:49.361503 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22252e0c803e4ef9b530c9e5c9d8a299" candidate_uuid: "edc78a35bff341cb936994588b93bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" is_pre_election: true
I20260924 19:12:49.362082 15752 raft_consensus.cc:2454] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc78a35bff341cb936994588b93bf2f in term 0.
I20260924 19:12:49.362084 15885 raft_consensus.cc:2454] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc78a35bff341cb936994588b93bf2f in term 0.
I20260924 19:12:49.362836 15554 leader_election.cc:302] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d45fafcc6776405897ca07a819269d98, edc78a35bff341cb936994588b93bf2f; no voters:
I20260924 19:12:49.363212 15956 raft_consensus.cc:2781] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:49.363338 15956 raft_consensus.cc:482] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:49.363478 15956 raft_consensus.cc:3037] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:49.365712 15956 raft_consensus.cc:504] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.366344 15956 leader_election.cc:288] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [CANDIDATE]: Term 1 election: Requested vote from peers a2d459686fc74baeaeb8d5b17d5482bc (127.11.18.66:46411), d45fafcc6776405897ca07a819269d98 (127.11.18.67:42345)
I20260924 19:12:49.366874 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22252e0c803e4ef9b530c9e5c9d8a299" candidate_uuid: "edc78a35bff341cb936994588b93bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d459686fc74baeaeb8d5b17d5482bc"
I20260924 19:12:49.366904 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22252e0c803e4ef9b530c9e5c9d8a299" candidate_uuid: "edc78a35bff341cb936994588b93bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45fafcc6776405897ca07a819269d98"
I20260924 19:12:49.367080 15752 raft_consensus.cc:3037] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:49.367123 15885 raft_consensus.cc:3037] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:49.369383 15752 raft_consensus.cc:2454] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc78a35bff341cb936994588b93bf2f in term 1.
I20260924 19:12:49.369737 15885 raft_consensus.cc:2454] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc78a35bff341cb936994588b93bf2f in term 1.
I20260924 19:12:49.369875 15553 leader_election.cc:302] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2d459686fc74baeaeb8d5b17d5482bc, edc78a35bff341cb936994588b93bf2f; no voters:
I20260924 19:12:49.370219 15956 raft_consensus.cc:2781] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:49.370704 15956 raft_consensus.cc:686] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 1 LEADER]: Becoming Leader. State: Replica: edc78a35bff341cb936994588b93bf2f, State: Running, Role: LEADER
I20260924 19:12:49.371279 15956 consensus_queue.cc:237] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } }
I20260924 19:12:49.377594 15475 catalog_manager.cc:5495] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f reported cstate change: term changed from 0 to 1, leader changed from <none> to edc78a35bff341cb936994588b93bf2f (127.11.18.65). New cstate: current_term: 1 leader_uuid: "edc78a35bff341cb936994588b93bf2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc78a35bff341cb936994588b93bf2f" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 41173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 } health_report { overall_health: UNKNOWN } } }
W20260924 19:12:49.385943 15798 tablet.cc:2369] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:12:49.397907 15931 tablet.cc:2369] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:12:49.416651 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.410588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.11.18.65:41173)
0924 19:12:49.410786 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.410790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.410809 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:12:49.411028 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.411032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.411127 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.411255 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:49.411788 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.412616 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.414259 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.414675 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.414793 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.415370 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:49.415886 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:49.415998 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.416238 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.416353 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:12:49.425419 15752 raft_consensus.cc:1261] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 1 FOLLOWER]: Refusing update from remote peer edc78a35bff341cb936994588b93bf2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:49.425447 15885 raft_consensus.cc:1261] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 1 FOLLOWER]: Refusing update from remote peer edc78a35bff341cb936994588b93bf2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:49.426611 15959 consensus_queue.cc:1035] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d459686fc74baeaeb8d5b17d5482bc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:49.427035 15956 consensus_queue.cc:1035] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45fafcc6776405897ca07a819269d98" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:49.447264 15964 mvcc.cc:205] Tried to move back new op lower bound from 7332975285955727360 to 7332975285749415936. Current Snapshot: MvccSnapshot[applied={T|T < 7332975285955727360}]
I20260924 19:12:49.467837 15475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:49.468163 15475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:49.472010 15475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9355ec1550fd4b6aa356c6faff7d6253: Sending DeleteTablet for 3 replicas of tablet 22252e0c803e4ef9b530c9e5c9d8a299
I20260924 19:12:49.473393 15732 tablet_service.cc:1514] Processing DeleteTablet for tablet 22252e0c803e4ef9b530c9e5c9d8a299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:49 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:47358
I20260924 19:12:49.473640 15599 tablet_service.cc:1514] Processing DeleteTablet for tablet 22252e0c803e4ef9b530c9e5c9d8a299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:49 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:33632
I20260924 19:12:49.474202 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 22252e0c803e4ef9b530c9e5c9d8a299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:49 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:35680
I20260924 19:12:49.474602 15972 tablet_replica.cc:330] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: stopping tablet replica
I20260924 19:12:49.474970 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 15537
I20260924 19:12:49.475230 15972 raft_consensus.cc:2229] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:49.475952 15972 raft_consensus.cc:2258] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:49.477553 15973 tablet_replica.cc:330] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: stopping tablet replica
I20260924 19:12:49.478027 15973 raft_consensus.cc:2229] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:12:49.478826 15973 raft_consensus.cc:2258] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:49.481153 15973 ts_tablet_manager.cc:1905] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:49.481161 15972 ts_tablet_manager.cc:1905] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:49.485818 15974 tablet_replica.cc:330] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: stopping tablet replica
I20260924 19:12:49.486354 15974 raft_consensus.cc:2229] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:49.486948 15974 raft_consensus.cc:2258] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 19:12:49 dist-test-slave-0lgr krb5kdc[15432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:12:49.488868 15972 ts_tablet_manager.cc:1918] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:12:49.489045 15972 log.cc:1196] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/22252e0c803e4ef9b530c9e5c9d8a299
I20260924 19:12:49.489174 15973 ts_tablet_manager.cc:1918] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:12:49.489431 15973 log.cc:1196] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/22252e0c803e4ef9b530c9e5c9d8a299
I20260924 19:12:49.489537 15974 ts_tablet_manager.cc:1905] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:49.489645 15972 ts_tablet_manager.cc:1939] T 22252e0c803e4ef9b530c9e5c9d8a299 P a2d459686fc74baeaeb8d5b17d5482bc: Deleting consensus metadata
I20260924 19:12:49.490199 15973 ts_tablet_manager.cc:1939] T 22252e0c803e4ef9b530c9e5c9d8a299 P edc78a35bff341cb936994588b93bf2f: Deleting consensus metadata
I20260924 19:12:49.491639 15461 catalog_manager.cc:4841] TS a2d459686fc74baeaeb8d5b17d5482bc (127.11.18.66:46411): tablet 22252e0c803e4ef9b530c9e5c9d8a299 (table test-table [id=d4f6c189188c46a69946ccfce4a69103]) successfully deleted
I20260924 19:12:49.491940 15460 catalog_manager.cc:4841] TS edc78a35bff341cb936994588b93bf2f (127.11.18.65:41173): tablet 22252e0c803e4ef9b530c9e5c9d8a299 (table test-table [id=d4f6c189188c46a69946ccfce4a69103]) successfully deleted
I20260924 19:12:49.497573 15974 ts_tablet_manager.cc:1918] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:12:49.497769 15974 log.cc:1196] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/22252e0c803e4ef9b530c9e5c9d8a299
I20260924 19:12:49.498375 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.476845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.11.18.65:41173)
0924 19:12:49.477026 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.477033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.477054 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:12:49.477123 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.477130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.477232 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.477385 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:49.479214 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.480860 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.483052 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.483925 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.484042 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.490220 (+ 6178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:49.490238 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:49.490258 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:49.490319 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:49.494768 (+ 4449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.495745 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.495755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.495759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.495958 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.496427 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.496433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.496437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.496910 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:49.497213 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.497594 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.497873 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:12:49.498387 15974 ts_tablet_manager.cc:1939] T 22252e0c803e4ef9b530c9e5c9d8a299 P d45fafcc6776405897ca07a819269d98: Deleting consensus metadata
I20260924 19:12:49.499138 15659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:49.500433 15462 catalog_manager.cc:4841] TS d45fafcc6776405897ca07a819269d98 (127.11.18.67:42345): tablet 22252e0c803e4ef9b530c9e5c9d8a299 (table test-table [id=d4f6c189188c46a69946ccfce4a69103]) successfully deleted
I20260924 19:12:49.598065 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 15537
I20260924 19:12:49.616694 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 15670
Sep 24 19:12:49 dist-test-slave-0lgr krb5kdc[15432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:12:49.633975 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.617638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.11.18.66:46411)
0924 19:12:49.617826 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.617830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.617864 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:12:49.618072 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.618077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.618147 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.618253 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:49.619538 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.620345 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.621895 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.622341 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.622447 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.627233 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:49.627250 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:49.627266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:49.627310 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:49.630426 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.631266 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.631275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.631277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.631424 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.632024 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.632029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.632031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.632507 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:49.632932 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.633295 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.633597 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:12:49.634760 15792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:49.731912 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 15670
I20260924 19:12:49.745929 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 15803
Sep 24 19:12:49 dist-test-slave-0lgr krb5kdc[15432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:12:49.761926 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:49.746779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.11.18.67:42345)
0924 19:12:49.746975 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:49.746981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:49.747053 (+ 72us) server_negotiation.cc:407] Connection header received
0924 19:12:49.747259 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:49.747265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:49.747341 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:49.747449 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:49.748815 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.749758 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.751109 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:49.751506 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:49.751644 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:49.756049 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:49.756065 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:49.756081 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:49.756127 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:49.759155 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.759827 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.759834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.759836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.759971 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:49.760349 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:49.760355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:49.760359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:49.760711 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:49.760995 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:49.761377 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 19:12:49.761591 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:12:49.762696 15925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:49.851193 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 15803
I20260924 19:12:49.867835 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 15445
I20260924 19:12:49.869029 15506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:49.985054 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 15445
2026-09-24T19:12:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4406 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T19:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:50Z Disabled control of system clock
I20260924 19:12:50.058804 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38893
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:38893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:12:50.412052 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:50.412380 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:50.412474 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:50.420893 15983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:50.421033 15983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:50.421077 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:50.421131 15983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:50.421185 15983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:50.431820 15983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:38893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:50.433265 15983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:50.435213 15983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:50.449225 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:50.449250 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:50.449306 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:50.450184 15983 server_base.cc:1034] running on GCE node
I20260924 19:12:50.451002 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:50.452633 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:50.453879 15983 hybrid_clock.cc:648] HybridClock initialized: now 1790277170453778 us; error 98 us; skew 500 ppm
I20260924 19:12:50.454221 15983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:12:50.456740 15983 webserver.cc:460] Webserver started at http://127.11.18.126:41363/ using document root <none> and password file <none>
I20260924 19:12:50.457525 15983 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:50.457628 15983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:50.457887 15983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:50.460774 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "1792b6e56b92419db73a8942374450a5"
format_stamp: "Formatted at 2026-09-24 19:12:50 on dist-test-slave-0lgr"
server_key: "0290e708165637ae878f0a2f420ce364"
server_key_iv: "075cd6c7789804a36ad28c916ad72868"
server_key_version: "encryptionkey@0"
I20260924 19:12:50.461668 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "1792b6e56b92419db73a8942374450a5"
format_stamp: "Formatted at 2026-09-24 19:12:50 on dist-test-slave-0lgr"
server_key: "0290e708165637ae878f0a2f420ce364"
server_key_iv: "075cd6c7789804a36ad28c916ad72868"
server_key_version: "encryptionkey@0"
I20260924 19:12:50.467118 15983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:12:50.470783 15997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:50.472352 15983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:50.472556 15983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "1792b6e56b92419db73a8942374450a5"
format_stamp: "Formatted at 2026-09-24 19:12:50 on dist-test-slave-0lgr"
server_key: "0290e708165637ae878f0a2f420ce364"
server_key_iv: "075cd6c7789804a36ad28c916ad72868"
server_key_version: "encryptionkey@0"
I20260924 19:12:50.472795 15983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:50.478708 15983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:50.479528 15983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:50.479844 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:50.499692 15983 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:38893
I20260924 19:12:50.499755 16048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:38893 every 8 connection(s)
I20260924 19:12:50.501224 15983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:50.505302 16049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:50.510898 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 15983
I20260924 19:12:50.511145 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:50.511855 11337 external_mini_cluster.cc:1428] Setting key 28bacd223c7c1d84ada520056826c94e
I20260924 19:12:50.517531 16049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5: Bootstrap starting.
I20260924 19:12:50.521749 16049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:50.522274 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:50.514325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.11.18.126:38893)
0924 19:12:50.515467 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:50.515483 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:50.515535 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:12:50.517893 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:50.517921 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:50.518469 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:50.518917 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:50.520029 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:50.520038 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:50.520054 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:50.520072 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:50.520148 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:50.520179 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:50.520391 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:50.520515 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:12:50.520794 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":324,"threads_started":1}
I20260924 19:12:50.523208 16049 log.cc:824] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:50.526458 16049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5: No bootstrap required, opened a new log
I20260924 19:12:50.535030 16049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792b6e56b92419db73a8942374450a5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38893 } }
I20260924 19:12:50.535492 16049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:50.535620 16049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1792b6e56b92419db73a8942374450a5, State: Initialized, Role: FOLLOWER
I20260924 19:12:50.536405 16049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792b6e56b92419db73a8942374450a5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38893 } }
I20260924 19:12:50.536769 16049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:50.536942 16049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:50.537128 16049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:50.539757 16049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792b6e56b92419db73a8942374450a5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38893 } }
I20260924 19:12:50.540314 16049 leader_election.cc:302] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1792b6e56b92419db73a8942374450a5; no voters:
I20260924 19:12:50.540938 16049 leader_election.cc:288] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:50.541126 16054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:50.541989 16054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [term 1 LEADER]: Becoming Leader. State: Replica: 1792b6e56b92419db73a8942374450a5, State: Running, Role: LEADER
I20260924 19:12:50.542577 16054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792b6e56b92419db73a8942374450a5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38893 } }
I20260924 19:12:50.543268 16049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:50.545779 16056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1792b6e56b92419db73a8942374450a5. Latest consensus state: current_term: 1 leader_uuid: "1792b6e56b92419db73a8942374450a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792b6e56b92419db73a8942374450a5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38893 } } }
I20260924 19:12:50.546159 16056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:50.549976 16055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1792b6e56b92419db73a8942374450a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1792b6e56b92419db73a8942374450a5" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38893 } } }
I20260924 19:12:50.550598 16055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:50.554551 16062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:50.559849 16062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:50.571399 16062 catalog_manager.cc:1347] Generated new cluster ID: af5fff3ddedb45a58f602aa824b195ec
I20260924 19:12:50.571630 16062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:50.595651 16062 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:50.596092 16062 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:50.620859 16062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5: Generated new TSK 0
I20260924 19:12:50.621891 16062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:12:50.627817 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38893
--builtin_ntp_servers=127.11.18.84:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:12:50.987608 16073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:50.987923 16073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:50.988020 16073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:50.996398 16073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:50.996518 16073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:50.996650 16073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:12:51.008757 16073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:51.010198 16073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:51.012019 16073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:51.026317 16078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:51.026453 16079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:51.026535 16081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:51.028527 16073 server_base.cc:1034] running on GCE node
I20260924 19:12:51.029282 16073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:51.030436 16073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:51.031713 16073 hybrid_clock.cc:648] HybridClock initialized: now 1790277171031663 us; error 40 us; skew 500 ppm
I20260924 19:12:51.032130 16073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:12:51.034943 16073 webserver.cc:460] Webserver started at http://127.11.18.65:41999/ using document root <none> and password file <none>
I20260924 19:12:51.035732 16073 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:51.035887 16073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:51.036273 16073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:51.039531 16073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "df4e9b98b86f4df3823796308b4063ef"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "a84b55ce4950d37497fbcf363ed8a469"
server_key_iv: "51e8149f33538fd7052a3336c337b898"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.040453 16073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "df4e9b98b86f4df3823796308b4063ef"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "a84b55ce4950d37497fbcf363ed8a469"
server_key_iv: "51e8149f33538fd7052a3336c337b898"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.046365 16073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:12:51.050163 16087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:51.051857 16073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:51.052042 16073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "df4e9b98b86f4df3823796308b4063ef"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "a84b55ce4950d37497fbcf363ed8a469"
server_key_iv: "51e8149f33538fd7052a3336c337b898"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.052338 16073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:51.058324 16073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:51.059296 16073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:51.059585 16073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:51.060601 16073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:51.062283 16073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:51.062407 16073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:51.062547 16073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:51.062634 16073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:51.098031 16073 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:33293
I20260924 19:12:51.098116 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:33293 every 8 connection(s)
I20260924 19:12:51.099403 16073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:12:51.103833 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 16073
I20260924 19:12:51.104024 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:12:51.104799 11337 external_mini_cluster.cc:1428] Setting key 82617fe4637af95ebdd1e51c14f28e43
I20260924 19:12:51.108012 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38893
--builtin_ntp_servers=127.11.18.84:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 19:12:51.110176 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:51.102461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:56373 (local address 127.11.18.126:38893)
0924 19:12:51.103050 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:51.103056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:51.104502 (+ 1446us) server_negotiation.cc:407] Connection header received
0924 19:12:51.106913 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:51.106917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:51.107013 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:51.107120 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:51.109022 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:51.109032 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:51.109040 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:51.109066 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:51.109123 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:51.109142 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:51.109275 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:51.109710 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 19:12:51.109781 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":260,"threads_started":1}
I20260924 19:12:51.111145 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:51.103102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38893 (local address 127.11.18.65:56373)
0924 19:12:51.104303 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:51.104347 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:51.106565 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:51.107440 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:51.107450 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:51.107648 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:51.108646 (+ 998us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:12:51.108649 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:51.108689 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:12:51.108799 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:51.109295 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:51.109306 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:51.109313 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:12:51.109768 (+ 455us) client_negotiation.cc:240] Negotiation successful
0924 19:12:51.110015 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":448,"threads_started":1}
I20260924 19:12:51.113178 16200 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38893
I20260924 19:12:51.113526 16200 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:51.114271 16200 heartbeater.cc:511] Master 127.11.18.126:38893 requested a full tablet report, sending...
I20260924 19:12:51.116458 16014 ts_manager.cc:194] Registered new tserver with Master: df4e9b98b86f4df3823796308b4063ef (127.11.18.65:33293)
W20260924 19:12:51.443192 16204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:51.443547 16204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:51.443715 16204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:51.451249 16204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:51.451454 16204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:51.451653 16204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:12:51.461555 16204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:51.462960 16204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:51.464740 16204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:51.478881 16209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:51.478914 16210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:51.479110 16212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:51.480207 16204 server_base.cc:1034] running on GCE node
I20260924 19:12:51.480858 16204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:51.481957 16204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:51.483206 16204 hybrid_clock.cc:648] HybridClock initialized: now 1790277171483141 us; error 33 us; skew 500 ppm
I20260924 19:12:51.483567 16204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:12:51.486716 16204 webserver.cc:460] Webserver started at http://127.11.18.66:39621/ using document root <none> and password file <none>
I20260924 19:12:51.487673 16204 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:51.487783 16204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:51.488215 16204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:51.491923 16204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "4e7688ecf35e4e2b9449021e1618270e"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "8d3f02bcd8c74c6ec967fec832b44e0c"
server_key_iv: "18e2ec3bfe896b275ded6e3cd55cd954"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.492872 16204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "4e7688ecf35e4e2b9449021e1618270e"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "8d3f02bcd8c74c6ec967fec832b44e0c"
server_key_iv: "18e2ec3bfe896b275ded6e3cd55cd954"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.499200 16204 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 19:12:51.503062 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:51.504594 16204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:12:51.504848 16204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "4e7688ecf35e4e2b9449021e1618270e"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "8d3f02bcd8c74c6ec967fec832b44e0c"
server_key_iv: "18e2ec3bfe896b275ded6e3cd55cd954"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.505100 16204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:51.511423 16204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:51.512337 16204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:51.512645 16204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:51.513749 16204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:51.515283 16204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:51.515415 16204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:51.515554 16204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:51.515651 16204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:51.552464 16204 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:38937
I20260924 19:12:51.552567 16330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:38937 every 8 connection(s)
I20260924 19:12:51.554032 16204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:12:51.560488 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 16204
I20260924 19:12:51.560714 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:12:51.561239 11337 external_mini_cluster.cc:1428] Setting key a7152896f2ed6644e34dd4e2189e6426
I20260924 19:12:51.563174 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:51.556990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:58271 (local address 127.11.18.126:38893)
0924 19:12:51.557170 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:51.557175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:51.558745 (+ 1570us) server_negotiation.cc:407] Connection header received
0924 19:12:51.560506 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:51.560512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:51.560584 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:51.560684 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:51.562283 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:51.562288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:51.562292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:51.562309 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:51.562343 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:51.562388 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:51.562533 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:51.562775 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 19:12:51.562838 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:12:51.563977 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:51.557551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38893 (local address 127.11.18.66:58271)
0924 19:12:51.558563 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:51.558602 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:51.560216 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:51.560964 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:51.560973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:51.561076 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:51.561867 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:12:51.561871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:51.561925 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:12:51.562049 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:51.562517 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:51.562526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:51.562532 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:12:51.562754 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 19:12:51.562954 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":311,"threads_started":1}
I20260924 19:12:51.564515 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38893
--builtin_ntp_servers=127.11.18.84:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 19:12:51.565671 16331 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38893
I20260924 19:12:51.565968 16331 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:51.566565 16331 heartbeater.cc:511] Master 127.11.18.126:38893 requested a full tablet report, sending...
I20260924 19:12:51.568058 16014 ts_manager.cc:194] Registered new tserver with Master: 4e7688ecf35e4e2b9449021e1618270e (127.11.18.66:38937)
W20260924 19:12:51.900266 16334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:51.900561 16334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:51.900660 16334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:51.908464 16334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:51.908605 16334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:51.908732 16334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:12:51.919236 16334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:38893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:51.920539 16334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:51.922331 16334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:51.936209 16339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:51.936312 16342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:51.936553 16340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:51.937440 16334 server_base.cc:1034] running on GCE node
I20260924 19:12:51.938107 16334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:51.939131 16334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:51.940353 16334 hybrid_clock.cc:648] HybridClock initialized: now 1790277171940306 us; error 40 us; skew 500 ppm
I20260924 19:12:51.940764 16334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:12:51.943555 16334 webserver.cc:460] Webserver started at http://127.11.18.67:45977/ using document root <none> and password file <none>
I20260924 19:12:51.944319 16334 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:51.944427 16334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:51.944814 16334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:51.947937 16334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "f05ba97a17c343539a51db1af3ca31c7"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "a2420e7f831064035971ddb4b461c39c"
server_key_iv: "ac4242abe08517df6c3b71861509c18a"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.948815 16334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "f05ba97a17c343539a51db1af3ca31c7"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "a2420e7f831064035971ddb4b461c39c"
server_key_iv: "ac4242abe08517df6c3b71861509c18a"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.955065 16334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:12:51.960552 16348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:51.962249 16334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:12:51.962424 16334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "f05ba97a17c343539a51db1af3ca31c7"
format_stamp: "Formatted at 2026-09-24 19:12:51 on dist-test-slave-0lgr"
server_key: "a2420e7f831064035971ddb4b461c39c"
server_key_iv: "ac4242abe08517df6c3b71861509c18a"
server_key_version: "encryptionkey@0"
I20260924 19:12:51.962666 16334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:51.968616 16334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:51.969472 16334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:51.969745 16334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:51.970678 16334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:12:51.972079 16334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:12:51.972167 16334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:51.972321 16334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:12:51.972420 16334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:52.009243 16334 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:35867
I20260924 19:12:52.009377 16460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:35867 every 8 connection(s)
I20260924 19:12:52.010762 16334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:12:52.016675 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 16334
I20260924 19:12:52.016878 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:12:52.017552 11337 external_mini_cluster.cc:1428] Setting key 88682455a93a4e29735bf79e9e4be9b6
I20260924 19:12:52.019816 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.013550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:41885 (local address 127.11.18.126:38893)
0924 19:12:52.013772 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.013778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.015417 (+ 1639us) server_negotiation.cc:407] Connection header received
0924 19:12:52.017162 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.017169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.017254 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.017443 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.019055 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.019061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.019066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.019090 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.019142 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.019160 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.019291 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.019474 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.019537 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 19:12:52.020884 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.014173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:38893 (local address 127.11.18.67:41885)
0924 19:12:52.015254 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.015297 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:52.016794 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:52.017615 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:52.017624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:52.017805 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:52.018602 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:12:52.018605 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:52.018711 (+ 106us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:12:52.018830 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:52.019267 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:52.019275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:52.019282 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:12:52.019501 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 19:12:52.019832 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":437,"threads_started":1}
I20260924 19:12:52.022454 16461 heartbeater.cc:347] Connected to a master server at 127.11.18.126:38893
I20260924 19:12:52.022845 16461 heartbeater.cc:464] Registering TS with master...
I20260924 19:12:52.023443 16461 heartbeater.cc:511] Master 127.11.18.126:38893 requested a full tablet report, sending...
I20260924 19:12:52.024880 16014 ts_manager.cc:194] Registered new tserver with Master: f05ba97a17c343539a51db1af3ca31c7 (127.11.18.67:35867)
I20260924 19:12:52.033787 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:12:52.041308 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.038544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.11.18.126:38893)
0924 19:12:52.038779 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.038784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.039238 (+ 454us) server_negotiation.cc:407] Connection header received
0924 19:12:52.039366 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.039370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.039425 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.039527 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.040622 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.040627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.040679 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.040696 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.040730 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.040743 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.040855 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.041023 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.041080 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 19:12:52.045970 16014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:12:52.053576 16014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:12:52.076937 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.067491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.11.18.66:38937)
0924 19:12:52.069317 (+ 1826us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.069366 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.072143 (+ 2777us) server_negotiation.cc:407] Connection header received
0924 19:12:52.072555 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.072569 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.072903 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.073069 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.075915 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.075925 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.075940 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.075966 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.076020 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.076039 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.076174 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.076495 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.076594 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1560,"thread_start_us":1397,"threads_started":1}
I20260924 19:12:52.076987 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.067985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.11.18.67:35867)
0924 19:12:52.069913 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.069922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.072703 (+ 2781us) server_negotiation.cc:407] Connection header received
0924 19:12:52.072971 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.072979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.073206 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.073375 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.075915 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.075925 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.075940 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.075961 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.076020 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.076040 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.076182 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.076491 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.076594 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1743,"spinlock_wait_cycles":26368,"thread_start_us":1709,"threads_started":1}
I20260924 19:12:52.077626 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.069896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.11.18.65:33293)
0924 19:12:52.070494 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.070506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.074248 (+ 3742us) server_negotiation.cc:407] Connection header received
0924 19:12:52.074363 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.074371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.074584 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.074757 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.076581 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.076591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.076604 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.076624 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.076688 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.076715 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.076834 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.077158 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.077278 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":307,"threads_started":1}
I20260924 19:12:52.078059 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.067277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:35867 (local address 127.0.0.1:35008)
0924 19:12:52.072585 (+ 5308us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.072602 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:52.072816 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:52.074438 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:52.074445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:52.074539 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:52.075681 (+ 1142us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:12:52.075683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:52.075696 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:12:52.075729 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:52.076280 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:52.076296 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:52.076303 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:12:52.077375 (+ 1072us) client_negotiation.cc:240] Negotiation successful
0924 19:12:52.077492 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5160,"thread_start_us":4096,"threads_started":1}
I20260924 19:12:52.078176 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.067222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:38937 (local address 127.0.0.1:50234)
0924 19:12:52.071935 (+ 4713us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.071999 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:52.072314 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:52.074483 (+ 2169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:52.074486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:52.074538 (+ 52us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:52.075578 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:12:52.075586 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:52.075623 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:12:52.075718 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:52.076299 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:52.076303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:52.076307 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:12:52.077400 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0924 19:12:52.077507 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4350,"mutex_wait_us":25,"thread_start_us":6605,"threads_started":1}
I20260924 19:12:52.078578 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.069697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36734)
0924 19:12:52.074101 (+ 4404us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.074119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:52.074224 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:52.074798 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:52.074809 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:52.074834 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:52.076330 (+ 1496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:12:52.076334 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:52.076349 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:12:52.076395 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:52.076978 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:52.076983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:52.076988 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:12:52.077988 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0924 19:12:52.078045 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4288,"thread_start_us":2597,"threads_started":1}
I20260924 19:12:52.082067 16266 tablet_service.cc:1467] Processing CreateTablet for tablet d7d0d8e228354b2094da017e8ea575ce (DEFAULT_TABLE table=test-table [id=8e0b578b8eab4f609e845fa276353eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:52.082070 16135 tablet_service.cc:1467] Processing CreateTablet for tablet d7d0d8e228354b2094da017e8ea575ce (DEFAULT_TABLE table=test-table [id=8e0b578b8eab4f609e845fa276353eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:52.082070 16396 tablet_service.cc:1467] Processing CreateTablet for tablet d7d0d8e228354b2094da017e8ea575ce (DEFAULT_TABLE table=test-table [id=8e0b578b8eab4f609e845fa276353eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:12:52.083478 16396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d0d8e228354b2094da017e8ea575ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:52.083470 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d0d8e228354b2094da017e8ea575ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:52.083871 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d0d8e228354b2094da017e8ea575ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:52.096503 16479 tablet_bootstrap.cc:492] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: Bootstrap starting.
I20260924 19:12:52.099481 16480 tablet_bootstrap.cc:492] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: Bootstrap starting.
I20260924 19:12:52.100587 16479 tablet_bootstrap.cc:654] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:52.103663 16479 log.cc:824] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:52.103737 16480 tablet_bootstrap.cc:654] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:52.105203 16480 log.cc:824] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:52.109283 16481 tablet_bootstrap.cc:492] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef: Bootstrap starting.
I20260924 19:12:52.112159 16480 tablet_bootstrap.cc:492] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: No bootstrap required, opened a new log
I20260924 19:12:52.112453 16480 ts_tablet_manager.cc:1397] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260924 19:12:52.112571 16479 tablet_bootstrap.cc:492] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: No bootstrap required, opened a new log
I20260924 19:12:52.112840 16479 ts_tablet_manager.cc:1397] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260924 19:12:52.115998 16481 tablet_bootstrap.cc:654] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:52.117067 16481 log.cc:824] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:52.119872 16200 heartbeater.cc:503] Master 127.11.18.126:38893 was elected leader, sending a full tablet report...
I20260924 19:12:52.123337 16480 raft_consensus.cc:348] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.123808 16480 raft_consensus.cc:374] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:52.123904 16480 raft_consensus.cc:729] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05ba97a17c343539a51db1af3ca31c7, State: Initialized, Role: FOLLOWER
I20260924 19:12:52.124056 16479 raft_consensus.cc:348] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.124481 16479 raft_consensus.cc:374] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:52.124574 16479 raft_consensus.cc:729] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7688ecf35e4e2b9449021e1618270e, State: Initialized, Role: FOLLOWER
I20260924 19:12:52.124696 16480 consensus_queue.cc:260] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.125159 16479 consensus_queue.cc:260] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.127444 16331 heartbeater.cc:503] Master 127.11.18.126:38893 was elected leader, sending a full tablet report...
I20260924 19:12:52.127986 16461 heartbeater.cc:503] Master 127.11.18.126:38893 was elected leader, sending a full tablet report...
I20260924 19:12:52.128302 16479 ts_tablet_manager.cc:1428] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 19:12:52.129036 16481 tablet_bootstrap.cc:492] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef: No bootstrap required, opened a new log
I20260924 19:12:52.129384 16481 ts_tablet_manager.cc:1397] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20260924 19:12:52.129966 16480 ts_tablet_manager.cc:1428] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20260924 19:12:52.130504 16485 raft_consensus.cc:482] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:12:52.130839 16485 raft_consensus.cc:504] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.132979 16485 leader_election.cc:288] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e7688ecf35e4e2b9449021e1618270e (127.11.18.66:38937), df4e9b98b86f4df3823796308b4063ef (127.11.18.65:33293)
I20260924 19:12:52.136240 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.133344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:38937 (local address 127.11.18.67:38053)
0924 19:12:52.133743 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.133763 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:52.133852 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:52.134496 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:52.134504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:52.134540 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:52.135246 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:12:52.135251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:52.135277 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:12:52.135339 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:52.135775 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:52.135780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:52.135786 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:12:52.135886 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:12:52.135942 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260924 19:12:52.136826 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.133707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:38053 (local address 127.11.18.66:38937)
0924 19:12:52.134017 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.134024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.134045 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:12:52.134105 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.134114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.134202 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.134320 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.135527 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.135534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.135539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.135561 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.135598 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.135614 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.135724 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.136463 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.136527 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260924 19:12:52.137835 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.133814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:38185 (local address 127.11.18.65:33293)
0924 19:12:52.134083 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.134089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.135339 (+ 1250us) server_negotiation.cc:407] Connection header received
0924 19:12:52.135570 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.135577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.135655 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.135729 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.136993 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.137000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.137005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.137025 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.137075 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.137093 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.137194 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.137455 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.137522 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 19:12:52.137732 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d0d8e228354b2094da017e8ea575ce" candidate_uuid: "f05ba97a17c343539a51db1af3ca31c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7688ecf35e4e2b9449021e1618270e" is_pre_election: true
I20260924 19:12:52.138238 16286 raft_consensus.cc:2454] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f05ba97a17c343539a51db1af3ca31c7 in term 0.
I20260924 19:12:52.138582 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.134179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.11.18.67:38185)
0924 19:12:52.135234 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.135257 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:52.135423 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:52.135937 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:52.135942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:52.135968 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:52.136742 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:12:52.136748 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:52.136767 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:12:52.136817 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:52.137288 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:52.137293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:52.137298 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:12:52.138064 (+ 766us) client_negotiation.cc:240] Negotiation successful
0924 19:12:52.138122 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":973,"threads_started":1}
I20260924 19:12:52.139019 16350 leader_election.cc:302] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7688ecf35e4e2b9449021e1618270e, f05ba97a17c343539a51db1af3ca31c7; no voters:
I20260924 19:12:52.139454 16485 raft_consensus.cc:2781] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:12:52.139605 16485 raft_consensus.cc:482] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:12:52.139725 16485 raft_consensus.cc:3037] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:52.139829 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d0d8e228354b2094da017e8ea575ce" candidate_uuid: "f05ba97a17c343539a51db1af3ca31c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4e9b98b86f4df3823796308b4063ef" is_pre_election: true
W20260924 19:12:52.141470 16349 leader_election.cc:341] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df4e9b98b86f4df3823796308b4063ef (127.11.18.65:33293): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:12:52.142815 16485 raft_consensus.cc:504] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.142977 16481 raft_consensus.cc:348] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.143563 16481 raft_consensus.cc:374] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:52.143639 16485 leader_election.cc:288] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [CANDIDATE]: Term 1 election: Requested vote from peers 4e7688ecf35e4e2b9449021e1618270e (127.11.18.66:38937), df4e9b98b86f4df3823796308b4063ef (127.11.18.65:33293)
I20260924 19:12:52.143817 16481 raft_consensus.cc:729] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4e9b98b86f4df3823796308b4063ef, State: Initialized, Role: FOLLOWER
I20260924 19:12:52.144253 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d0d8e228354b2094da017e8ea575ce" candidate_uuid: "f05ba97a17c343539a51db1af3ca31c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4e9b98b86f4df3823796308b4063ef"
I20260924 19:12:52.144254 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d0d8e228354b2094da017e8ea575ce" candidate_uuid: "f05ba97a17c343539a51db1af3ca31c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7688ecf35e4e2b9449021e1618270e"
I20260924 19:12:52.144505 16286 raft_consensus.cc:3037] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:52.144685 16481 consensus_queue.cc:260] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.145216 16155 raft_consensus.cc:3037] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:52.147066 16481 ts_tablet_manager.cc:1428] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20260924 19:12:52.147457 16286 raft_consensus.cc:2454] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f05ba97a17c343539a51db1af3ca31c7 in term 1.
I20260924 19:12:52.148015 16350 leader_election.cc:302] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7688ecf35e4e2b9449021e1618270e, f05ba97a17c343539a51db1af3ca31c7; no voters:
I20260924 19:12:52.148386 16485 raft_consensus.cc:2781] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:52.148900 16155 raft_consensus.cc:2454] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f05ba97a17c343539a51db1af3ca31c7 in term 1.
I20260924 19:12:52.149099 16485 raft_consensus.cc:686] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 1 LEADER]: Becoming Leader. State: Replica: f05ba97a17c343539a51db1af3ca31c7, State: Running, Role: LEADER
I20260924 19:12:52.149852 16485 consensus_queue.cc:237] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } }
I20260924 19:12:52.155771 16014 catalog_manager.cc:5495] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f05ba97a17c343539a51db1af3ca31c7 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "f05ba97a17c343539a51db1af3ca31c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f05ba97a17c343539a51db1af3ca31c7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 35867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 } health_report { overall_health: UNKNOWN } } }
I20260924 19:12:52.182044 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.179600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.11.18.67:35867)
0924 19:12:52.179833 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.179839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.179948 (+ 109us) server_negotiation.cc:407] Connection header received
0924 19:12:52.180216 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.180220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.180300 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.180414 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.181430 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.181435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.181438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.181453 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.181484 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.181501 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.181633 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.181737 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.181795 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 19:12:52.190591 16155 raft_consensus.cc:1261] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [term 1 FOLLOWER]: Refusing update from remote peer f05ba97a17c343539a51db1af3ca31c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:52.190712 16286 raft_consensus.cc:1261] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 1 FOLLOWER]: Refusing update from remote peer f05ba97a17c343539a51db1af3ca31c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:52.191593 16489 consensus_queue.cc:1035] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7688ecf35e4e2b9449021e1618270e" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:52.191975 16485 consensus_queue.cc:1035] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4e9b98b86f4df3823796308b4063ef" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:52.221271 16014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:52.221599 16014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:12:52.225499 16014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1792b6e56b92419db73a8942374450a5: Sending DeleteTablet for 3 replicas of tablet d7d0d8e228354b2094da017e8ea575ce
I20260924 19:12:52.226652 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d0d8e228354b2094da017e8ea575ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:52 UTC) from {username='slave'} at 127.0.0.1:50234
I20260924 19:12:52.226701 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d0d8e228354b2094da017e8ea575ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:52 UTC) from {username='slave'} at 127.0.0.1:36734
I20260924 19:12:52.226754 16396 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d0d8e228354b2094da017e8ea575ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:12:52 UTC) from {username='slave'} at 127.0.0.1:35008
I20260924 19:12:52.234977 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 16073
I20260924 19:12:52.237867 16504 tablet_replica.cc:330] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: stopping tablet replica
I20260924 19:12:52.238351 16504 raft_consensus.cc:2229] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:12:52.239130 16504 raft_consensus.cc:2258] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:52.240787 16503 tablet_replica.cc:330] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef: stopping tablet replica
I20260924 19:12:52.242093 16502 tablet_replica.cc:330] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: stopping tablet replica
I20260924 19:12:52.242584 16502 raft_consensus.cc:2229] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:52.243177 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.237649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.11.18.65:33293)
0924 19:12:52.237879 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.237887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.239988 (+ 2101us) server_negotiation.cc:407] Connection header received
0924 19:12:52.240209 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.240217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.240371 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.240513 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.241985 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.241991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.241995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.242015 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.242052 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.242067 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.242742 (+ 675us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.242802 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.242861 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:12:52.243798 16504 ts_tablet_manager.cc:1905] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:52.244041 16195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:52.241240 16503 raft_consensus.cc:2229] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:12:52.251717 16502 raft_consensus.cc:2258] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:52.254370 16502 ts_tablet_manager.cc:1905] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:12:52.254635 16504 ts_tablet_manager.cc:1918] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:12:52.254905 16504 log.cc:1196] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/d7d0d8e228354b2094da017e8ea575ce
I20260924 19:12:52.255570 16504 ts_tablet_manager.cc:1939] T d7d0d8e228354b2094da017e8ea575ce P f05ba97a17c343539a51db1af3ca31c7: Deleting consensus metadata
I20260924 19:12:52.257952 16001 catalog_manager.cc:4841] TS f05ba97a17c343539a51db1af3ca31c7 (127.11.18.67:35867): tablet d7d0d8e228354b2094da017e8ea575ce (table test-table [id=8e0b578b8eab4f609e845fa276353eed]) successfully deleted
I20260924 19:12:52.263484 16502 ts_tablet_manager.cc:1918] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:12:52.263845 16502 log.cc:1196] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/d7d0d8e228354b2094da017e8ea575ce
I20260924 19:12:52.264454 16502 ts_tablet_manager.cc:1939] T d7d0d8e228354b2094da017e8ea575ce P 4e7688ecf35e4e2b9449021e1618270e: Deleting consensus metadata
I20260924 19:12:52.266463 15999 catalog_manager.cc:4841] TS 4e7688ecf35e4e2b9449021e1618270e (127.11.18.66:38937): tablet d7d0d8e228354b2094da017e8ea575ce (table test-table [id=8e0b578b8eab4f609e845fa276353eed]) successfully deleted
I20260924 19:12:52.356827 16503 raft_consensus.cc:2258] T d7d0d8e228354b2094da017e8ea575ce P df4e9b98b86f4df3823796308b4063ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:12:52.357087 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 16073
I20260924 19:12:52.379580 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 16204
W20260924 19:12:52.379595 15998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.18.65:33293 (error 108)
I20260924 19:12:52.383441 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.382882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36744)
0924 19:12:52.383077 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.383251 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 19:12:52.383649 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.380613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.11.18.66:38937)
0924 19:12:52.381284 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.381290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.381312 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:12:52.381404 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.381410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.381519 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.381633 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.382887 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.382894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.382899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.382921 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.382960 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.382975 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.383109 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.383294 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.383355 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527}
W20260924 19:12:52.384189 15998 catalog_manager.cc:4584] TS df4e9b98b86f4df3823796308b4063ef (127.11.18.65:33293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7d0d8e228354b2094da017e8ea575ce: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
I20260924 19:12:52.384457 16326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:52.437969 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.437433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36760)
0924 19:12:52.437667 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.437774 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 19:12:52.439129 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.438730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36764)
0924 19:12:52.438893 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.438994 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:12:52.476245 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 16204
I20260924 19:12:52.493009 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 16334
I20260924 19:12:52.496646 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.493919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.11.18.67:35867)
0924 19:12:52.494188 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:52.494196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:52.494225 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:12:52.494299 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:52.494304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:52.494376 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:52.494664 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:52.495993 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:52.496000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:52.496005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:52.496024 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:52.496063 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:52.496081 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:52.496234 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:52.496279 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 19:12:52.496339 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:12:52.497570 16456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:52.499919 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.499405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36774)
0924 19:12:52.499587 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.499741 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:12:52.500901 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.500576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36778)
0924 19:12:52.500710 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.500776 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 19:12:52.581648 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.581169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36794)
0924 19:12:52.581415 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.581509 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:12:52.582662 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.582284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36796)
0924 19:12:52.582490 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.582554 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 19:12:52.630227 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 16334
I20260924 19:12:52.651017 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 15983
I20260924 19:12:52.652774 16044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:52.766310 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:52.765631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33293 (local address 127.0.0.1:36810)
0924 19:12:52.765990 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:52.766117 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260924 19:12:52.766316 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 15983
2026-09-24T19:12:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2778 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 19:12:52.809674 11337 mini_oidc.cc:152] Starting JWKS server
I20260924 19:12:52.812402 11337 webserver.cc:460] Webserver started at https://127.0.0.1:38109/ using document root <none> and password file <none>
I20260924 19:12:52.812752 11337 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:12:52.813650 11337 webserver.cc:460] Webserver started at http://127.0.0.1:42841/ using document root <none> and password file <none>
2026-09-24T19:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:52Z Disabled control of system clock
I20260924 19:12:52.842057 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45231
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45231
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38109/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/testchainca.cert with env {}
W20260924 19:12:53.180689 16516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:53.181073 16516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:53.181166 16516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:53.188454 16516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:53.188580 16516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:53.188618 16516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:53.188668 16516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:53.188712 16516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:12:53.188773 16516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:12:53.188813 16516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38109/jwks.json
I20260924 19:12:53.198966 16516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38109/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:53.200224 16516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:53.201588 16516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:53.213954 16524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:53.214007 16521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:53.213973 16522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:53.214985 16516 server_base.cc:1034] running on GCE node
I20260924 19:12:53.215749 16516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:53.217136 16516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:53.218372 16516 hybrid_clock.cc:648] HybridClock initialized: now 1790277173218307 us; error 49 us; skew 500 ppm
I20260924 19:12:53.218760 16516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:12:53.221141 16516 webserver.cc:460] Webserver started at http://127.11.18.126:46489/ using document root <none> and password file <none>
I20260924 19:12:53.221949 16516 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:53.222041 16516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:53.222288 16516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:53.225219 16516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "5e5469f84cc44b499464067778204be4"
format_stamp: "Formatted at 2026-09-24 19:12:53 on dist-test-slave-0lgr"
server_key: "013a1baa09aab809a2dbc6da4bcc817e"
server_key_iv: "5275b829f5194933898e8eaf571f3da8"
server_key_version: "encryptionkey@0"
I20260924 19:12:53.226050 16516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "5e5469f84cc44b499464067778204be4"
format_stamp: "Formatted at 2026-09-24 19:12:53 on dist-test-slave-0lgr"
server_key: "013a1baa09aab809a2dbc6da4bcc817e"
server_key_iv: "5275b829f5194933898e8eaf571f3da8"
server_key_version: "encryptionkey@0"
I20260924 19:12:53.231048 16516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:12:53.234452 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:53.235889 16516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:12:53.236052 16516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "5e5469f84cc44b499464067778204be4"
format_stamp: "Formatted at 2026-09-24 19:12:53 on dist-test-slave-0lgr"
server_key: "013a1baa09aab809a2dbc6da4bcc817e"
server_key_iv: "5275b829f5194933898e8eaf571f3da8"
server_key_version: "encryptionkey@0"
I20260924 19:12:53.236246 16516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:53.253566 16516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:53.254935 16516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:53.255304 16516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:53.279330 16516 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:45231
I20260924 19:12:53.279264 16581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:45231 every 8 connection(s)
I20260924 19:12:53.280880 16516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:53.283057 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 16516
I20260924 19:12:53.283324 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:53.284029 11337 external_mini_cluster.cc:1428] Setting key 2b1031802380922388f1ecf061e6ab54
I20260924 19:12:53.285279 16582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:53.300539 16582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4: Bootstrap starting.
I20260924 19:12:53.301007 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:53.286110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.11.18.126:45231)
0924 19:12:53.287162 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:53.287173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:53.287216 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:12:53.289737 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:53.289764 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:53.290268 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:53.290686 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:53.291373 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:53.293998 (+ 2625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:53.295877 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:53.296668 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:53.296853 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:53.298332 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:53.298339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:53.298355 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:53.299018 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:53.299104 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:53.299130 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:53.299229 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:53.299521 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:12:53.299955 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":401,"threads_started":1}
I20260924 19:12:53.304266 16582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:53.305305 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:12:53.305627 16582 log.cc:824] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:53.308663 16582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4: No bootstrap required, opened a new log
I20260924 19:12:53.316260 16582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5469f84cc44b499464067778204be4" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45231 } }
I20260924 19:12:53.316659 16582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:53.316728 16582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5469f84cc44b499464067778204be4, State: Initialized, Role: FOLLOWER
I20260924 19:12:53.317304 16582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5469f84cc44b499464067778204be4" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45231 } }
I20260924 19:12:53.317636 16582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:53.317821 16582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:53.318043 16582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:53.320304 16582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5469f84cc44b499464067778204be4" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45231 } }
I20260924 19:12:53.320816 16582 leader_election.cc:302] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e5469f84cc44b499464067778204be4; no voters:
I20260924 19:12:53.321486 16582 leader_election.cc:288] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:53.321702 16587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:53.322517 16587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5469f84cc44b499464067778204be4, State: Running, Role: LEADER
I20260924 19:12:53.323000 16587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5469f84cc44b499464067778204be4" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45231 } }
I20260924 19:12:53.323776 16582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:53.325799 16588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e5469f84cc44b499464067778204be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5469f84cc44b499464067778204be4" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45231 } } }
I20260924 19:12:53.326222 16588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:53.326547 16589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5469f84cc44b499464067778204be4. Latest consensus state: current_term: 1 leader_uuid: "5e5469f84cc44b499464067778204be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5469f84cc44b499464067778204be4" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45231 } } }
I20260924 19:12:53.326850 16589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:53.328687 16592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:53.333956 16592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:53.343537 16592 catalog_manager.cc:1347] Generated new cluster ID: e29d76f53d8b4ed789a5e3ba8d964ad1
I20260924 19:12:53.343676 16592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:53.362001 16592 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:53.363929 16592 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:53.372756 16592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e5469f84cc44b499464067778204be4: Generated new TSK 0
I20260924 19:12:53.373872 16592 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:12:53.426548 16612 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 19:12:53.426734 16612 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:12:53.427917 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:53.404393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.11.18.126:45231)
0924 19:12:53.404611 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:53.404618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:53.405165 (+ 547us) server_negotiation.cc:407] Connection header received
0924 19:12:53.405540 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:53.405548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:53.405656 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:53.405824 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:12:53.406404 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:53.407600 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:53.409530 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:53.410303 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:53.410474 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:53.411188 (+ 714us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:12:53.427199 (+ 16011us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:12:53.427587 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50088: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 19:12:53.428035 16585 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50088: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 19:12:53.429525 16611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:53.404334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45231 (local address 127.0.0.1:50088)
0924 19:12:53.405055 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:53.405074 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:53.405239 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:53.405902 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:53.405906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:53.405916 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:12:53.406254 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:53.406264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:53.407924 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:53.407929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:53.409199 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:53.409214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:53.410615 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:53.410622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:53.410770 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:53.410843 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:12:53.427507 (+ 16664us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:12:53.427868 (+ 361us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.18.126:45231: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":382,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.18.126:45231: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 19:12:53.431483 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 16516
I20260924 19:12:53.432778 16577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:53.534057 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 16516
2026-09-24T19:12:53Z chronyd exiting
I20260924 19:12:53.585170 11337 test_util.cc:175] -----------------------------------------------
I20260924 19:12:53.585310 11337 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277128821086-11337-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (780 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 19:12:53.589095 11337 mini_oidc.cc:152] Starting JWKS server
I20260924 19:12:53.591015 11337 webserver.cc:460] Webserver started at https://127.0.0.1:38599/ using document root <none> and password file <none>
I20260924 19:12:53.591245 11337 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:12:53.591918 11337 webserver.cc:460] Webserver started at http://127.0.0.1:46215/ using document root <none> and password file <none>
2026-09-24T19:12:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:53Z Disabled control of system clock
I20260924 19:12:53.619721 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36697
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:36697
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38599/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/testchainca.cert with env {}
W20260924 19:12:53.943410 16619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:53.943739 16619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:53.943890 16619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:53.951342 16619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:53.951471 16619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:53.951510 16619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:53.951629 16619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:53.951699 16619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:12:53.951788 16619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:12:53.951848 16619 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38599/jwks.json
I20260924 19:12:53.962387 16619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:36697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38599/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:53.963967 16619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:53.965353 16619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:53.977653 16624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:53.977778 16627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:53.977775 16625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:53.983316 16619 server_base.cc:1034] running on GCE node
I20260924 19:12:53.984248 16619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:53.985900 16619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:53.987138 16619 hybrid_clock.cc:648] HybridClock initialized: now 1790277173987089 us; error 46 us; skew 500 ppm
I20260924 19:12:53.987496 16619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:12:53.993983 16619 webserver.cc:460] Webserver started at http://127.11.18.126:41499/ using document root <none> and password file <none>
I20260924 19:12:53.994747 16619 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:53.994891 16619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:53.995237 16619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:53.998217 16619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "52d4af4a802846458e0f3a874f9c4a09"
format_stamp: "Formatted at 2026-09-24 19:12:53 on dist-test-slave-0lgr"
server_key: "8f782a43af6e9792acf03ef2e523e1da"
server_key_iv: "f1f3766a9da603b5c2bf994b90c80af7"
server_key_version: "encryptionkey@0"
I20260924 19:12:53.999051 16619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "52d4af4a802846458e0f3a874f9c4a09"
format_stamp: "Formatted at 2026-09-24 19:12:53 on dist-test-slave-0lgr"
server_key: "8f782a43af6e9792acf03ef2e523e1da"
server_key_iv: "f1f3766a9da603b5c2bf994b90c80af7"
server_key_version: "encryptionkey@0"
I20260924 19:12:54.004310 16619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:12:54.008087 16633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:54.009804 16619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:54.010022 16619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "52d4af4a802846458e0f3a874f9c4a09"
format_stamp: "Formatted at 2026-09-24 19:12:53 on dist-test-slave-0lgr"
server_key: "8f782a43af6e9792acf03ef2e523e1da"
server_key_iv: "f1f3766a9da603b5c2bf994b90c80af7"
server_key_version: "encryptionkey@0"
I20260924 19:12:54.010263 16619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:54.028927 16619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:54.029902 16619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:54.030226 16619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:54.049750 16619 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:36697
I20260924 19:12:54.049769 16684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:36697 every 8 connection(s)
I20260924 19:12:54.051172 16619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:54.055249 16685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:54.059952 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 16619
I20260924 19:12:54.060158 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:54.060825 11337 external_mini_cluster.cc:1428] Setting key a55200698544bdb886da14d8cf09cbf0
I20260924 19:12:54.068146 16685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09: Bootstrap starting.
I20260924 19:12:54.072383 16685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:54.073800 16685 log.cc:824] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:54.077517 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:54.063049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.11.18.126:36697)
0924 19:12:54.064339 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:54.064352 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:54.064399 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:12:54.066489 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:54.066516 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:54.067049 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:54.067414 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:54.068100 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.070106 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.072156 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.072959 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.073102 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:54.074800 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:54.074819 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:54.074841 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:54.075648 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:54.075735 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:54.075757 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:54.075850 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:54.076134 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 19:12:54.076483 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":575,"threads_started":1}
I20260924 19:12:54.077673 16685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09: No bootstrap required, opened a new log
I20260924 19:12:54.081772 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:12:54.086391 16685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d4af4a802846458e0f3a874f9c4a09" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36697 } }
I20260924 19:12:54.086824 16685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:54.086933 16685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d4af4a802846458e0f3a874f9c4a09, State: Initialized, Role: FOLLOWER
I20260924 19:12:54.087721 16685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d4af4a802846458e0f3a874f9c4a09" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36697 } }
I20260924 19:12:54.087981 16685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:54.088131 16685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:54.088326 16685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:54.091046 16685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d4af4a802846458e0f3a874f9c4a09" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36697 } }
I20260924 19:12:54.091583 16685 leader_election.cc:302] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d4af4a802846458e0f3a874f9c4a09; no voters:
I20260924 19:12:54.092182 16685 leader_election.cc:288] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:54.092362 16690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:54.093116 16690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [term 1 LEADER]: Becoming Leader. State: Replica: 52d4af4a802846458e0f3a874f9c4a09, State: Running, Role: LEADER
I20260924 19:12:54.093881 16690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d4af4a802846458e0f3a874f9c4a09" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36697 } }
I20260924 19:12:54.094594 16685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:54.097088 16691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d4af4a802846458e0f3a874f9c4a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d4af4a802846458e0f3a874f9c4a09" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36697 } } }
I20260924 19:12:54.097065 16692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d4af4a802846458e0f3a874f9c4a09. Latest consensus state: current_term: 1 leader_uuid: "52d4af4a802846458e0f3a874f9c4a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d4af4a802846458e0f3a874f9c4a09" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36697 } } }
I20260924 19:12:54.097537 16691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:54.097571 16692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:54.100374 16697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:54.106196 16697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:54.115684 16697 catalog_manager.cc:1347] Generated new cluster ID: 0b8048e4ff4c41399a248ca5713f3ee7
I20260924 19:12:54.115823 16697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:54.141810 16697 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:54.143159 16697 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:54.152429 16697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52d4af4a802846458e0f3a874f9c4a09: Generated new TSK 0
I20260924 19:12:54.153501 16697 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:12:54.198088 16715 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 19:12:54.198307 16715 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:12:54.199682 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:54.176957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.11.18.126:36697)
0924 19:12:54.177163 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:54.177170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:54.177791 (+ 621us) server_negotiation.cc:407] Connection header received
0924 19:12:54.178013 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:54.178019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:54.178111 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:54.178297 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:12:54.178868 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.180001 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.181659 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.182341 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.182485 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:54.183055 (+ 570us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:12:54.198734 (+ 15679us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:12:54.199298 (+ 564us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56264: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 19:12:54.199721 16714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:54.176862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36697 (local address 127.0.0.1:56264)
0924 19:12:54.177685 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:54.177701 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:54.177831 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:54.178358 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:54.178362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:54.178373 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:12:54.178697 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:54.178707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.180170 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.180174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:54.181503 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:54.181516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.182567 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.182572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:54.182723 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:54.182762 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:12:54.199053 (+ 16291us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:12:54.199334 (+ 281us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.18.126:36697: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":451,"threads_started":1}
W20260924 19:12:54.199822 16688 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56264: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 19:12:54.201647 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 16619
I20260924 19:12:54.202944 16680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:54.303529 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 16619
2026-09-24T19:12:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (770 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 19:12:54.360531 11337 mini_oidc.cc:152] Starting JWKS server
I20260924 19:12:54.362676 11337 webserver.cc:460] Webserver started at https://127.0.0.1:32817/ using document root <none> and password file <none>
I20260924 19:12:54.362927 11337 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:12:54.363619 11337 webserver.cc:460] Webserver started at http://127.0.0.1:39587/ using document root <none> and password file <none>
2026-09-24T19:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:54Z Disabled control of system clock
I20260924 19:12:54.391431 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41337
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:41337
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32817/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:12:54.729550 16722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:54.729880 16722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:54.730037 16722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:54.737248 16722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:54.737428 16722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:54.737528 16722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:54.737641 16722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:54.737712 16722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:12:54.737820 16722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:12:54.737892 16722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32817/jwks.json
I20260924 19:12:54.747246 16722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:41337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32817/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:54.748680 16722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:54.750552 16722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:54.763689 16727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:54.763779 16728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:54.763733 16730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:54.764781 16722 server_base.cc:1034] running on GCE node
I20260924 19:12:54.765792 16722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:54.767378 16722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:54.768776 16722 hybrid_clock.cc:648] HybridClock initialized: now 1790277174768721 us; error 39 us; skew 500 ppm
I20260924 19:12:54.769166 16722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:12:54.771705 16722 webserver.cc:460] Webserver started at http://127.11.18.126:41131/ using document root <none> and password file <none>
I20260924 19:12:54.772418 16722 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:54.772504 16722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:54.772831 16722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:54.775789 16722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "ae518636bc314df9888d3570f6eda80c"
format_stamp: "Formatted at 2026-09-24 19:12:54 on dist-test-slave-0lgr"
server_key: "76c2aef4d7ad047e0c4ed5e04195be77"
server_key_iv: "df8b86f701b25429f31c02f18b2f1a60"
server_key_version: "encryptionkey@0"
I20260924 19:12:54.776626 16722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "ae518636bc314df9888d3570f6eda80c"
format_stamp: "Formatted at 2026-09-24 19:12:54 on dist-test-slave-0lgr"
server_key: "76c2aef4d7ad047e0c4ed5e04195be77"
server_key_iv: "df8b86f701b25429f31c02f18b2f1a60"
server_key_version: "encryptionkey@0"
I20260924 19:12:54.782147 16722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:12:54.786008 16736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:54.787693 16722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260924 19:12:54.787909 16722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "ae518636bc314df9888d3570f6eda80c"
format_stamp: "Formatted at 2026-09-24 19:12:54 on dist-test-slave-0lgr"
server_key: "76c2aef4d7ad047e0c4ed5e04195be77"
server_key_iv: "df8b86f701b25429f31c02f18b2f1a60"
server_key_version: "encryptionkey@0"
I20260924 19:12:54.788152 16722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:54.802984 16722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:54.803933 16722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:54.804292 16722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:54.825011 16722 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:41337
I20260924 19:12:54.825037 16787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:41337 every 8 connection(s)
I20260924 19:12:54.826555 16722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:54.830415 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:54.831637 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 16722
I20260924 19:12:54.831863 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:54.832703 11337 external_mini_cluster.cc:1428] Setting key 5ce884defd872e542664ffca6bbf945d
I20260924 19:12:54.846449 16788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c: Bootstrap starting.
I20260924 19:12:54.848048 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:54.834845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.11.18.126:41337)
0924 19:12:54.836202 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:54.836213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:54.836264 (+ 51us) server_negotiation.cc:407] Connection header received
0924 19:12:54.838161 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:54.838185 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:54.838743 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:54.839111 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:54.839759 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.841165 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.843330 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.844144 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.844291 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:54.845654 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:54.845662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:54.845677 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:54.846255 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:54.846331 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:54.846352 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:54.846448 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:54.846767 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 19:12:54.847103 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":452,"threads_started":1}
I20260924 19:12:54.850080 16788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:54.851375 16788 log.cc:824] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:54.852432 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:12:54.854628 16788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c: No bootstrap required, opened a new log
I20260924 19:12:54.864411 16788 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae518636bc314df9888d3570f6eda80c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41337 } }
I20260924 19:12:54.864917 16788 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:54.865031 16788 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae518636bc314df9888d3570f6eda80c, State: Initialized, Role: FOLLOWER
I20260924 19:12:54.865713 16788 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae518636bc314df9888d3570f6eda80c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41337 } }
I20260924 19:12:54.865919 16788 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:54.866012 16788 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:54.866135 16788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:54.868854 16788 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae518636bc314df9888d3570f6eda80c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41337 } }
I20260924 19:12:54.869536 16788 leader_election.cc:302] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae518636bc314df9888d3570f6eda80c; no voters:
I20260924 19:12:54.870113 16788 leader_election.cc:288] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:54.870285 16793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:54.870962 16793 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [term 1 LEADER]: Becoming Leader. State: Replica: ae518636bc314df9888d3570f6eda80c, State: Running, Role: LEADER
I20260924 19:12:54.871606 16793 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae518636bc314df9888d3570f6eda80c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41337 } }
I20260924 19:12:54.872908 16788 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:54.874708 16794 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae518636bc314df9888d3570f6eda80c. Latest consensus state: current_term: 1 leader_uuid: "ae518636bc314df9888d3570f6eda80c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae518636bc314df9888d3570f6eda80c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41337 } } }
I20260924 19:12:54.875219 16794 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:54.874696 16795 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae518636bc314df9888d3570f6eda80c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae518636bc314df9888d3570f6eda80c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41337 } } }
I20260924 19:12:54.877703 16795 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:54.878422 16799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:54.883853 16799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:54.892843 16799 catalog_manager.cc:1347] Generated new cluster ID: 7f0c24f568354baba030f2b6c89131b8
I20260924 19:12:54.892967 16799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:54.903355 16799 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:54.905293 16799 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:54.915297 16799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae518636bc314df9888d3570f6eda80c: Generated new TSK 0
I20260924 19:12:54.916433 16799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:12:54.940235 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:54.930763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.11.18.126:41337)
0924 19:12:54.930934 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:54.930938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:54.931387 (+ 449us) server_negotiation.cc:407] Connection header received
0924 19:12:54.931600 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:54.931607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:54.931700 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:54.931881 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:12:54.932341 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.933542 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.935109 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.935788 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.935937 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:54.936503 (+ 566us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:12:54.939489 (+ 2986us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32817/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:12:54.939884 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32817/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 19:12:54.940392 16791 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32817/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 19:12:54.941910 16817 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:54.930616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41337 (local address 127.0.0.1:56242)
0924 19:12:54.931267 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:54.931284 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:54.931414 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:54.931898 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:54.931901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:54.931917 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:12:54.932187 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:54.932195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.933732 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.933737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:54.934941 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:54.934954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:54.936011 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:54.936019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:54.936152 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:54.936189 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:12:54.939754 (+ 3565us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32817/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:12:54.941466 (+ 1712us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.18.126:41337: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32817/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":1327,"thread_start_us":338,"threads_started":1}
I20260924 19:12:54.944056 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 16722
I20260924 19:12:54.945457 16783 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 19:12:54.953078 16818 webserver.cc:533] Webserver: sslize failed:
E20260924 19:12:54.953192 16818 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:12:55.041760 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 16722
2026-09-24T19:12:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (737 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T19:12:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:55Z Disabled control of system clock
I20260924 19:12:55.126238 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:40925
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:37857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:40925
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/testchainca.cert with env {}
W20260924 19:12:55.465700 16823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:55.466024 16823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:55.466118 16823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:55.474372 16823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:55.474516 16823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:55.474560 16823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:55.474622 16823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:55.474669 16823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:12:55.474751 16823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:12:55.474819 16823 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 19:12:55.485684 16823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:40925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:40925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:55.487030 16823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:55.488276 16823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:55.500672 16828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:55.500716 16829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:55.500743 16831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:55.501639 16823 server_base.cc:1034] running on GCE node
I20260924 19:12:55.502471 16823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:55.504037 16823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:55.505282 16823 hybrid_clock.cc:648] HybridClock initialized: now 1790277175505240 us; error 40 us; skew 500 ppm
I20260924 19:12:55.505723 16823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:12:55.508270 16823 webserver.cc:460] Webserver started at http://127.11.18.126:41165/ using document root <none> and password file <none>
I20260924 19:12:55.509016 16823 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:55.509109 16823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:55.509470 16823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:55.512415 16823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "20631a827be54f0d9c62b1412da291c6"
format_stamp: "Formatted at 2026-09-24 19:12:55 on dist-test-slave-0lgr"
server_key: "fafef0652352d8e66f80de068b311713"
server_key_iv: "611c5da40112dc132918aca456a6824c"
server_key_version: "encryptionkey@0"
I20260924 19:12:55.513248 16823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "20631a827be54f0d9c62b1412da291c6"
format_stamp: "Formatted at 2026-09-24 19:12:55 on dist-test-slave-0lgr"
server_key: "fafef0652352d8e66f80de068b311713"
server_key_iv: "611c5da40112dc132918aca456a6824c"
server_key_version: "encryptionkey@0"
I20260924 19:12:55.519443 16823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 19:12:55.523990 16837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:55.526108 16823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 19:12:55.526610 16823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "20631a827be54f0d9c62b1412da291c6"
format_stamp: "Formatted at 2026-09-24 19:12:55 on dist-test-slave-0lgr"
server_key: "fafef0652352d8e66f80de068b311713"
server_key_iv: "611c5da40112dc132918aca456a6824c"
server_key_version: "encryptionkey@0"
I20260924 19:12:55.526895 16823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:55.541805 16823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:55.542711 16823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:55.543038 16823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:55.563695 16823 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:40925
I20260924 19:12:55.563726 16888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:40925 every 8 connection(s)
I20260924 19:12:55.565189 16823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:55.567185 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 16823
I20260924 19:12:55.567411 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:55.568387 11337 external_mini_cluster.cc:1428] Setting key d0d4da4f0978f2cc45aaf42ca11b3d39
I20260924 19:12:55.570417 16889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:55.582231 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:55.570607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.11.18.126:40925)
0924 19:12:55.571727 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:55.571738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:55.571778 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:12:55.573387 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:55.573422 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:55.573926 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:55.574365 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:55.574950 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:55.576326 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:55.577746 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:55.578439 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:55.578640 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:55.579854 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:55.579862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:55.579876 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:12:55.580448 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:55.580522 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:12:55.580541 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:55.580683 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:55.580930 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 19:12:55.581239 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":430,"threads_started":1}
I20260924 19:12:55.583039 16889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6: Bootstrap starting.
I20260924 19:12:55.586566 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:12:55.586724 16889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:55.587857 16889 log.cc:824] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:55.592151 16889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6: No bootstrap required, opened a new log
I20260924 19:12:55.599821 16889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20631a827be54f0d9c62b1412da291c6" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40925 } }
I20260924 19:12:55.600183 16889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:55.600317 16889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20631a827be54f0d9c62b1412da291c6, State: Initialized, Role: FOLLOWER
I20260924 19:12:55.600955 16889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20631a827be54f0d9c62b1412da291c6" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40925 } }
I20260924 19:12:55.601182 16889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:55.601329 16889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:55.601544 16889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:55.604004 16889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20631a827be54f0d9c62b1412da291c6" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40925 } }
I20260924 19:12:55.604526 16889 leader_election.cc:302] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20631a827be54f0d9c62b1412da291c6; no voters:
I20260924 19:12:55.605119 16889 leader_election.cc:288] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:55.605311 16894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:55.606352 16894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [term 1 LEADER]: Becoming Leader. State: Replica: 20631a827be54f0d9c62b1412da291c6, State: Running, Role: LEADER
I20260924 19:12:55.606882 16894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20631a827be54f0d9c62b1412da291c6" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40925 } }
I20260924 19:12:55.607518 16889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:55.609860 16896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20631a827be54f0d9c62b1412da291c6. Latest consensus state: current_term: 1 leader_uuid: "20631a827be54f0d9c62b1412da291c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20631a827be54f0d9c62b1412da291c6" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40925 } } }
I20260924 19:12:55.610224 16896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:55.610793 16895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20631a827be54f0d9c62b1412da291c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20631a827be54f0d9c62b1412da291c6" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40925 } } }
I20260924 19:12:55.611115 16895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:55.612519 16900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:55.618219 16900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:55.629806 16911 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:12:55.635015 16900 catalog_manager.cc:1347] Generated new cluster ID: 7c063280c50c4e229b94313e4cfddc5f
I20260924 19:12:55.635147 16900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:55.679339 16900 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:55.681664 16900 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:55.695676 16900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20631a827be54f0d9c62b1412da291c6: Generated new TSK 0
I20260924 19:12:55.696669 16900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:12:55.858562 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:55.724605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.11.18.126:40925)
0924 19:12:55.724840 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:55.724849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:55.725484 (+ 635us) server_negotiation.cc:407] Connection header received
0924 19:12:55.725718 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:55.725722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:55.725794 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:55.725939 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:12:55.726495 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:55.727324 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:55.729461 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:55.729943 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:55.730052 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:55.730752 (+ 700us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:12:55.857968 (+127216us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:12:55.858303 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43102: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 19:12:55.858731 16892 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43102: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 19:12:55.859478 16918 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:55.724630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:40925 (local address 127.0.0.1:43102)
0924 19:12:55.725313 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:55.725364 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:55.725529 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:55.725985 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:55.725990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:55.726005 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:12:55.726302 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:55.726314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:55.727484 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:55.727490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:55.729249 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:55.729264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:55.730124 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:55.730131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:55.730313 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:55.730362 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:12:55.858290 (+127928us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:12:55.858674 (+ 384us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.18.126:40925: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":324,"threads_started":1}
I20260924 19:12:55.860663 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 16823
I20260924 19:12:55.862041 16884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:55.969797 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 16823
2026-09-24T19:12:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (908 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (431 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1602 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:12:58 dist-test-slave-0lgr krb5kdc[16934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:12:58 dist-test-slave-0lgr krb5kdc[16934](info): set up 2 sockets
Sep 24 19:12:58 dist-test-slave-0lgr krb5kdc[16934](info): commencing operation
krb5kdc: starting...
W20260924 19:12:59.178212 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:12:59 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:12:59.406379 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42835
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:42835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:59.740388 16947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:59.740684 16947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:59.740840 16947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:59.748099 16947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:59.748217 16947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:59.748297 16947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:59.748383 16947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:59.748451 16947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:59.758518 16947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:42835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:59.759910 16947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:59.761206 16947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:59.774057 16952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:59.774029 16953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:59.773984 16955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:59.774966 16947 server_base.cc:1034] running on GCE node
I20260924 19:12:59.775866 16947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:59.777577 16947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:59.778827 16947 hybrid_clock.cc:648] HybridClock initialized: now 1790277179778755 us; error 70 us; skew 500 ppm
I20260924 19:12:59.779208 16947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:59 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277179, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:59.786228 16947 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:12:59.787912 16947 webserver.cc:460] Webserver started at http://127.11.18.126:43527/ using document root <none> and password file <none>
I20260924 19:12:59.788632 16947 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:59.788781 16947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:59.789104 16947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:59.792441 16947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "e60dbd1a3b054665b80fdd3928f6745c"
format_stamp: "Formatted at 2026-09-24 19:12:59 on dist-test-slave-0lgr"
server_key: "88855384271f30841c11fa7f7833d243"
server_key_iv: "2e159c4b299f1cdbc4cf14102c2832fe"
server_key_version: "encryptionkey@0"
I20260924 19:12:59.793270 16947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "e60dbd1a3b054665b80fdd3928f6745c"
format_stamp: "Formatted at 2026-09-24 19:12:59 on dist-test-slave-0lgr"
server_key: "88855384271f30841c11fa7f7833d243"
server_key_iv: "2e159c4b299f1cdbc4cf14102c2832fe"
server_key_version: "encryptionkey@0"
I20260924 19:12:59.798640 16947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 19:12:59.802277 16962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:59.803665 16947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:12:59.803862 16947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "e60dbd1a3b054665b80fdd3928f6745c"
format_stamp: "Formatted at 2026-09-24 19:12:59 on dist-test-slave-0lgr"
server_key: "88855384271f30841c11fa7f7833d243"
server_key_iv: "2e159c4b299f1cdbc4cf14102c2832fe"
server_key_version: "encryptionkey@0"
I20260924 19:12:59.804070 16947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:59.824667 16947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:59.829974 16947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:59.830324 16947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:59.851541 16947 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:42835
I20260924 19:12:59.851572 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:42835 every 8 connection(s)
I20260924 19:12:59.853041 16947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:59.857219 17014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:59.858381 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 16947
I20260924 19:12:59.858621 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:12:59.859539 11337 external_mini_cluster.cc:1428] Setting key a2af79ae0d351aae363bd0555219f869
I20260924 19:12:59.873620 17014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c: Bootstrap starting.
Sep 24 19:12:59 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:12:59.877979 17014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:59.879311 17014 log.cc:824] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:59.882956 17014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c: No bootstrap required, opened a new log
I20260924 19:12:59.886256 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:59.861948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.11.18.126:42835)
0924 19:12:59.862790 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:59.862802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:59.862849 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:12:59.864618 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:59.864644 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:59.865096 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:59.865562 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:59.867512 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:59.868788 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:59.870880 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:59.871611 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:59.871757 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:59.877310 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:59.877320 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:59.877505 (+ 185us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:59.877559 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:59.881541 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:59.882382 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:59.882391 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:59.882400 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:59.882629 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:59.883131 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:59.883143 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:59.883146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:59.883756 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:59.884128 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:59.884603 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 19:12:59.885074 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":380,"threads_started":1}
I20260924 19:12:59.892247 17014 raft_consensus.cc:348] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60dbd1a3b054665b80fdd3928f6745c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42835 } }
I20260924 19:12:59.892652 17014 raft_consensus.cc:374] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:12:59.892765 17014 raft_consensus.cc:729] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60dbd1a3b054665b80fdd3928f6745c, State: Initialized, Role: FOLLOWER
I20260924 19:12:59.893615 17014 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60dbd1a3b054665b80fdd3928f6745c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42835 } }
I20260924 19:12:59.893904 17014 raft_consensus.cc:388] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:12:59.894053 17014 raft_consensus.cc:482] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:12:59.894258 17014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:59.896551 17014 raft_consensus.cc:504] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60dbd1a3b054665b80fdd3928f6745c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42835 } }
I20260924 19:12:59.897032 17014 leader_election.cc:302] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e60dbd1a3b054665b80fdd3928f6745c; no voters:
I20260924 19:12:59.897691 17014 leader_election.cc:288] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:12:59.897804 17019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:59.899205 17019 raft_consensus.cc:686] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [term 1 LEADER]: Becoming Leader. State: Replica: e60dbd1a3b054665b80fdd3928f6745c, State: Running, Role: LEADER
I20260924 19:12:59.899752 17019 consensus_queue.cc:237] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60dbd1a3b054665b80fdd3928f6745c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42835 } }
I20260924 19:12:59.899850 17014 sys_catalog.cc:564] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:59.903035 17020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e60dbd1a3b054665b80fdd3928f6745c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60dbd1a3b054665b80fdd3928f6745c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42835 } } }
I20260924 19:12:59.903362 17020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:59.904364 17028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:12:59.903709 17021 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e60dbd1a3b054665b80fdd3928f6745c. Latest consensus state: current_term: 1 leader_uuid: "e60dbd1a3b054665b80fdd3928f6745c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60dbd1a3b054665b80fdd3928f6745c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42835 } } }
I20260924 19:12:59.904630 17021 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:59.910713 17028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:59.920650 17028 catalog_manager.cc:1347] Generated new cluster ID: bf1f5409ec2547e281635967e11da535
I20260924 19:12:59.920851 17028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:59.943608 17028 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:59.945111 17028 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:59.954083 17028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e60dbd1a3b054665b80fdd3928f6745c: Generated new TSK 0
I20260924 19:12:59.955190 17028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:13:00.019236 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42835
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:00.384586 17040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:00.384930 17040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:00.385109 17040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:00.392812 17040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:00.392979 17040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:00.393153 17040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:00.403456 17040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:00.404774 17040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:00.406162 17040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:00.419948 17045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:00.420089 17048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:00.420113 17046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:00.421087 17040 server_base.cc:1034] running on GCE node
I20260924 19:13:00.421849 17040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:00.422901 17040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:00.424167 17040 hybrid_clock.cc:648] HybridClock initialized: now 1790277180424098 us; error 70 us; skew 500 ppm
I20260924 19:13:00.424537 17040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:00 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:00.429246 17040 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:13:00.431404 17040 webserver.cc:460] Webserver started at http://127.11.18.65:34979/ using document root <none> and password file <none>
I20260924 19:13:00.432189 17040 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:00.432341 17040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:00.432669 17040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:00.435830 17040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "69555efece174ca09b87164174c1e65d"
format_stamp: "Formatted at 2026-09-24 19:13:00 on dist-test-slave-0lgr"
server_key: "f510fe4de27927d2fdf1f41d0c224aca"
server_key_iv: "31a908a6c4e5e01ba776d379ba701de5"
server_key_version: "encryptionkey@0"
I20260924 19:13:00.436688 17040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "69555efece174ca09b87164174c1e65d"
format_stamp: "Formatted at 2026-09-24 19:13:00 on dist-test-slave-0lgr"
server_key: "f510fe4de27927d2fdf1f41d0c224aca"
server_key_iv: "31a908a6c4e5e01ba776d379ba701de5"
server_key_version: "encryptionkey@0"
I20260924 19:13:00.442473 17040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:13:00.446466 17055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:00.448283 17040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:13:00.448580 17040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "69555efece174ca09b87164174c1e65d"
format_stamp: "Formatted at 2026-09-24 19:13:00 on dist-test-slave-0lgr"
server_key: "f510fe4de27927d2fdf1f41d0c224aca"
server_key_iv: "31a908a6c4e5e01ba776d379ba701de5"
server_key_version: "encryptionkey@0"
I20260924 19:13:00.448827 17040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:00.476125 17040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:00.480758 17040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:00.481094 17040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:00.482194 17040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:00.483639 17040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:00.483757 17040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:00.483929 17040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:00.484011 17040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:00.520973 17040 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:35317
I20260924 19:13:00.521085 17167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:35317 every 8 connection(s)
I20260924 19:13:00.522512 17040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:00.524885 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 17040
I20260924 19:13:00.525108 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:13:00.526075 11337 external_mini_cluster.cc:1428] Setting key df3ad467c8530df8d7dbde37260860e0
Sep 24 19:13:00 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:13:00.546947 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:00.525449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:59233 (local address 127.11.18.126:42835)
0924 19:13:00.525911 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:00.525915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:00.527538 (+ 1623us) server_negotiation.cc:407] Connection header received
0924 19:13:00.529783 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:00.529790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:00.529902 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:00.530029 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:00.531641 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:00.532448 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:00.534719 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:00.535181 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:00.535296 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:00.540271 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:00.540277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:00.540282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:00.540318 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:00.543428 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:00.544317 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:00.544328 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:00.544330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:00.544459 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:00.544968 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:00.544972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:00.544975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:00.545262 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:00.545491 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:00.546313 (+ 822us) server_negotiation.cc:299] Negotiation successful
0924 19:13:00.546554 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
I20260924 19:13:00.548127 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:00.526226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42835 (local address 127.11.18.65:59233)
0924 19:13:00.527285 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:00.527413 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:00.529475 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:00.530280 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:00.530289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:00.530846 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:00.531458 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:00.531477 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:00.532637 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:00.532641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:00.534523 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:00.534543 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:00.535451 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:00.535461 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:00.535825 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:00.536764 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:00.536803 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:00.539975 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:00.543609 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:00.543621 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:00.543641 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:00.544148 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:00.544587 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:00.544592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:00.544596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:00.544817 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:00.545485 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:00.545494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:00.545896 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 19:13:00.546191 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 19:13:00.546806 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":431,"threads_started":1}
I20260924 19:13:00.550010 17168 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42835
I20260924 19:13:00.550314 17168 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:00.551031 17168 heartbeater.cc:511] Master 127.11.18.126:42835 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
I20260924 19:13:00.553615 16979 ts_manager.cc:194] Registered new tserver with Master: 69555efece174ca09b87164174c1e65d (127.11.18.65:35317)
I20260924 19:13:00.555135 16979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:59233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:13:00.578724 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42835
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:00.927549 17174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:00.927846 17174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:00.927983 17174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:00.936231 17174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:00.936362 17174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:00.936559 17174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:00.947141 17174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:00.948500 17174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:00.949884 17174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:00.963821 17180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:00.963754 17179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:00.963771 17182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:00.964936 17174 server_base.cc:1034] running on GCE node
I20260924 19:13:00.965721 17174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:00.966751 17174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:00.967985 17174 hybrid_clock.cc:648] HybridClock initialized: now 1790277180967930 us; error 40 us; skew 500 ppm
I20260924 19:13:00.968400 17174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:00 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:00.972183 17174 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:13:00.973919 17174 webserver.cc:460] Webserver started at http://127.11.18.66:36003/ using document root <none> and password file <none>
I20260924 19:13:00.974628 17174 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:00.974774 17174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:00.975086 17174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:00.978214 17174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "3501f17145a44599a7cc4f83aa3df254"
format_stamp: "Formatted at 2026-09-24 19:13:00 on dist-test-slave-0lgr"
server_key: "5c3884668ff8ebb1cf50aad4ab6bae6b"
server_key_iv: "9d6045a512ff03694f73d1f95877769c"
server_key_version: "encryptionkey@0"
I20260924 19:13:00.979029 17174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "3501f17145a44599a7cc4f83aa3df254"
format_stamp: "Formatted at 2026-09-24 19:13:00 on dist-test-slave-0lgr"
server_key: "5c3884668ff8ebb1cf50aad4ab6bae6b"
server_key_iv: "9d6045a512ff03694f73d1f95877769c"
server_key_version: "encryptionkey@0"
I20260924 19:13:00.984267 17174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:13:00.988396 17189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:00.990140 17174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:13:00.990338 17174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "3501f17145a44599a7cc4f83aa3df254"
format_stamp: "Formatted at 2026-09-24 19:13:00 on dist-test-slave-0lgr"
server_key: "5c3884668ff8ebb1cf50aad4ab6bae6b"
server_key_iv: "9d6045a512ff03694f73d1f95877769c"
server_key_version: "encryptionkey@0"
I20260924 19:13:00.990576 17174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:01.006892 17174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:01.011584 17174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:01.011912 17174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:01.012923 17174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:01.014453 17174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:01.014568 17174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:01.014725 17174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:01.014809 17174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:01.054466 17174 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:38363
I20260924 19:13:01.054642 17301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:38363 every 8 connection(s)
I20260924 19:13:01.055866 17174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:01.060807 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 17174
I20260924 19:13:01.060997 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:13:01.061578 11337 external_mini_cluster.cc:1428] Setting key 7612ae4ca5d2c19be57a80fe81418441
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:01.081996 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.058789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:53227 (local address 127.11.18.126:42835)
0924 19:13:01.059349 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.059357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.060624 (+ 1267us) server_negotiation.cc:407] Connection header received
0924 19:13:01.063361 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.063368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.063456 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.063583 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.065989 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.066768 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.069667 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.070101 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.070208 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.074445 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:01.074455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:01.074463 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:01.074511 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:01.078409 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.079292 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.079301 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.079307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.079473 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.079904 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.079912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.079925 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.080314 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:01.080476 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:01.081384 (+ 908us) server_negotiation.cc:299] Negotiation successful
0924 19:13:01.081625 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":317,"threads_started":1}
I20260924 19:13:01.083026 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.059481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42835 (local address 127.11.18.66:53227)
0924 19:13:01.060297 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:01.060377 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:01.063059 (+ 2682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:01.063847 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:01.063860 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:01.064750 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:01.065721 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:01.065753 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.066945 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.066954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:01.069485 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:01.069505 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.070361 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.070372 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:01.070759 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.071667 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:01.071692 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:01.074172 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:01.078582 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:01.078590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:01.078608 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:01.079138 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:01.079589 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:01.079593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:01.079595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:01.079775 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:01.080493 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:01.080504 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:01.080896 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 19:13:01.081250 (+ 354us) client_negotiation.cc:240] Negotiation successful
0924 19:13:01.081812 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:13:01.084630 17302 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42835
I20260924 19:13:01.084928 17302 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:01.085880 17302 heartbeater.cc:511] Master 127.11.18.126:42835 requested a full tablet report, sending...
I20260924 19:13:01.087558 16979 ts_manager.cc:194] Registered new tserver with Master: 3501f17145a44599a7cc4f83aa3df254 (127.11.18.66:38363)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:13:01.088718 16979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:53227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:13:01.117865 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42835
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:01.448993 17308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:01.449380 17308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:01.449540 17308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:01.457180 17308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:01.457378 17308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:01.457561 17308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:01.467585 17308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:01.469183 17308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:01.470696 17308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:01.484810 17313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:01.484730 17316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:01.484789 17314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:01.491254 17308 server_base.cc:1034] running on GCE node
I20260924 19:13:01.491989 17308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:01.493105 17308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:01.494397 17308 hybrid_clock.cc:648] HybridClock initialized: now 1790277181494317 us; error 60 us; skew 500 ppm
I20260924 19:13:01.494788 17308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:01.498184 17308 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:13:01.499944 17308 webserver.cc:460] Webserver started at http://127.11.18.67:43595/ using document root <none> and password file <none>
I20260924 19:13:01.500653 17308 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:01.500768 17308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:01.501129 17308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:01.504209 17308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "18c709ab36d249e5b6fbd55a568e7e50"
format_stamp: "Formatted at 2026-09-24 19:13:01 on dist-test-slave-0lgr"
server_key: "8bd67b65229a768fb9b6983944a9c87c"
server_key_iv: "fc8dda2c8cbd13786190434b5866df5c"
server_key_version: "encryptionkey@0"
I20260924 19:13:01.504945 17308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "18c709ab36d249e5b6fbd55a568e7e50"
format_stamp: "Formatted at 2026-09-24 19:13:01 on dist-test-slave-0lgr"
server_key: "8bd67b65229a768fb9b6983944a9c87c"
server_key_iv: "fc8dda2c8cbd13786190434b5866df5c"
server_key_version: "encryptionkey@0"
I20260924 19:13:01.510427 17308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:13:01.514364 17323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:01.515980 17308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:01.516122 17308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "18c709ab36d249e5b6fbd55a568e7e50"
format_stamp: "Formatted at 2026-09-24 19:13:01 on dist-test-slave-0lgr"
server_key: "8bd67b65229a768fb9b6983944a9c87c"
server_key_iv: "fc8dda2c8cbd13786190434b5866df5c"
server_key_version: "encryptionkey@0"
I20260924 19:13:01.516309 17308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:01.533073 17308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:01.537746 17308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:01.538000 17308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:01.538992 17308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:01.540347 17308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:01.540419 17308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:01.540501 17308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:01.540556 17308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:01.558573 17168 heartbeater.cc:503] Master 127.11.18.126:42835 was elected leader, sending a full tablet report...
I20260924 19:13:01.574569 17308 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:35437
I20260924 19:13:01.574610 17435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:35437 every 8 connection(s)
I20260924 19:13:01.575827 17308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:01.579747 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 17308
I20260924 19:13:01.579934 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:13:01.580714 11337 external_mini_cluster.cc:1428] Setting key a1fc514f08b05ca5939cb2136e83e256
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:01.598177 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.578858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:38815 (local address 127.11.18.126:42835)
0924 19:13:01.579159 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.579165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.580738 (+ 1573us) server_negotiation.cc:407] Connection header received
0924 19:13:01.582850 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.582855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.582980 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.583086 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.584685 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.585503 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.587065 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.587519 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.587651 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.591573 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:01.591580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:01.591585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:01.591628 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:01.594632 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.595571 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.595579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.595583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.595721 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.596199 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.596207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.596211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.596567 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:01.596736 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:01.597544 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 19:13:01.597792 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260924 19:13:01.599259 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.579679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42835 (local address 127.11.18.67:38815)
0924 19:13:01.580570 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:01.580617 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:01.582552 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:01.583326 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:01.583334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:01.583934 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:01.584523 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:01.584540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.585660 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.585664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:01.586918 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:01.586930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.587750 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.587756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:01.587989 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.588749 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:01.588773 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:01.591273 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:01.594838 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:01.594850 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:01.594878 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:01.595455 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:01.595851 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:01.595856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:01.595860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:01.596077 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:01.596718 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:01.596727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:01.597109 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 19:13:01.597469 (+ 360us) client_negotiation.cc:240] Negotiation successful
0924 19:13:01.598055 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":447,"threads_started":1}
I20260924 19:13:01.600780 17436 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42835
I20260924 19:13:01.601140 17436 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:01.601951 17436 heartbeater.cc:511] Master 127.11.18.126:42835 requested a full tablet report, sending...
I20260924 19:13:01.603462 16979 ts_manager.cc:194] Registered new tserver with Master: 18c709ab36d249e5b6fbd55a568e7e50 (127.11.18.67:35437)
I20260924 19:13:01.604317 16979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:38815
I20260924 19:13:01.610203 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/test-admin.keytab.
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:01.640928 11337 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 19:13:01.647439 17443 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:01.644067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.11.18.65:35317)
0924 19:13:01.645238 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.645246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.645275 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:13:01.645412 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.645419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.645648 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.645792 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.646948 (+ 1156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52180: BlockingRecv error: recv got EOF from 127.0.0.1:52180 (error 108)
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":885,"threads_started":1}
W20260924 19:13:01.647881 11337 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5cc)
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:01.650151 11337 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:01.652002 11337 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:13:01.669638 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.654032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.11.18.65:35317)
0924 19:13:01.654226 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.654231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.654465 (+ 234us) server_negotiation.cc:407] Connection header received
0924 19:13:01.654640 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.654644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.654705 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.654794 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.655856 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.656855 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.658213 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.658863 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.658976 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.663187 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:01.663200 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:01.663258 (+ 58us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:01.663296 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:01.666647 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.667412 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.667419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.667421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.667619 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.667985 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.667989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.667991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.668431 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:01.668668 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:01.669016 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 19:13:01.669257 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 19:13:01.676592 17443 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:01.674332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.11.18.65:35317)
0924 19:13:01.674571 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.674576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.675018 (+ 442us) server_negotiation.cc:407] Connection header received
0924 19:13:01.675078 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.675081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.675141 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.675227 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.676247 (+ 1020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52206: BlockingRecv error: recv got EOF from 127.0.0.1:52206 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260924 19:13:01.676870 11337 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:01.679162 11337 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:01.680688 11337 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:13:01.697892 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.683011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.11.18.65:35317)
0924 19:13:01.683226 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.683233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.683415 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:13:01.683653 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.683658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.683724 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.683841 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.684904 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.685751 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.687115 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.687606 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.687752 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.691862 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:01.691870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:01.691875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:01.691922 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:01.694925 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.695768 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.695772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.695775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.695935 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.696388 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.696393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.696395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.696720 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:01.696933 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:01.697459 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 19:13:01.697641 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260924 19:13:01.703826 17443 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:01.701756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.11.18.65:35317)
0924 19:13:01.701976 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.701983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.702258 (+ 275us) server_negotiation.cc:407] Connection header received
0924 19:13:01.702467 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.702471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.702607 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.702717 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.703569 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52224: BlockingRecv error: recv got EOF from 127.0.0.1:52224 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 19:13:01.704190 11337 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277128821086-11337-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:01.706255 11337 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:01.707964 11337 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:13:01.725097 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.710075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.11.18.65:35317)
0924 19:13:01.710329 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.710337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.710609 (+ 272us) server_negotiation.cc:407] Connection header received
0924 19:13:01.710826 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.710830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.710894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.711043 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.711995 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.712727 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.714089 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.714528 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.714667 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.718643 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:01.718649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:01.718653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:01.718693 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:01.722158 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.722941 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.722948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.722954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.723126 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.723537 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.723543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.723547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.723847 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:01.724207 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:01.724508 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 19:13:01.724737 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:13:01.726845 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 17040
I20260924 19:13:01.740628 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.728034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.11.18.65:35317)
0924 19:13:01.728234 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.728238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.728612 (+ 374us) server_negotiation.cc:407] Connection header received
0924 19:13:01.728793 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.728797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.728857 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.728955 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.730083 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.730765 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.732197 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.732699 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.732821 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.735115 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:01.735120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:01.735123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:01.735159 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:01.737900 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.738617 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.738621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.738623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.738747 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.739255 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.739259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.739261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.739541 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:01.739758 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:01.740183 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 19:13:01.740353 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:13:01.741300 17163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:01.829164 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 17040
I20260924 19:13:01.846640 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 17174
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:13:01.865495 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.847721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.11.18.66:38363)
0924 19:13:01.848260 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.848267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.848292 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:13:01.848349 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.848355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.848590 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.848724 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.849817 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.851213 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.852722 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.853846 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.853983 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.858284 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:01.858319 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:01.858343 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:01.858398 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:01.862120 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.862952 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.862961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.862963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.863150 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.863675 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.863680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.863682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.864097 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:01.864413 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:01.864712 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 19:13:01.865011 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":301,"threads_started":1}
I20260924 19:13:01.866945 17297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:01.955287 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 17174
I20260924 19:13:01.971349 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 17308
Sep 24 19:13:01 dist-test-slave-0lgr krb5kdc[16934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:13:01.988929 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:01.972545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.11.18.67:35437)
0924 19:13:01.973277 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:01.973284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:01.973312 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:13:01.973434 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:01.973441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:01.973677 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:01.973814 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:01.974995 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.976045 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.977554 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:01.978188 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:01.978324 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:01.982859 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:01.982876 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:01.982891 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:01.982937 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:01.986016 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.986877 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.986884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.986886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.987008 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:01.987415 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:01.987418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:01.987420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:01.987806 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:01.988029 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:01.988308 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:13:01.988617 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":400,"threads_started":1}
I20260924 19:13:01.990274 17431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:02.077689 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 17308
I20260924 19:13:02.093618 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 16947
I20260924 19:13:02.094842 17009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:02.199211 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 16947
2026-09-24T19:13:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4201 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:13:02 dist-test-slave-0lgr krb5kdc[17463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:13:02 dist-test-slave-0lgr krb5kdc[17463](info): set up 2 sockets
Sep 24 19:13:02 dist-test-slave-0lgr krb5kdc[17463](info): commencing operation
krb5kdc: starting...
W20260924 19:13:03.358166 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:13:03 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:13:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.126.keytab.
Entry for principal oryx/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.126.keytab.
I20260924 19:13:03.582093 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:34731
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.126.keytab
--principal=oryx/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:34731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:03.914297 17476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:03.914685 17476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:03.914863 17476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:03.922282 17476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:03.922459 17476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:03.922564 17476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:03.922665 17476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:03.922755 17476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:03.933300 17476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:34731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.126.keytab
--principal=oryx/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:34731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:03.934763 17476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:03.936098 17476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:03.948051 17481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:03.947961 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:03.947988 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:03.949069 17476 server_base.cc:1034] running on GCE node
I20260924 19:13:03.949975 17476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:03.951560 17476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:03.952806 17476 hybrid_clock.cc:648] HybridClock initialized: now 1790277183952730 us; error 71 us; skew 500 ppm
I20260924 19:13:03.953166 17476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:03 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:03.956724 17476 init.cc:377] Logged in from keytab as oryx/127.11.18.126@KRBTEST.COM (short username oryx)
I20260924 19:13:03.958422 17476 webserver.cc:460] Webserver started at http://127.11.18.126:37307/ using document root <none> and password file <none>
I20260924 19:13:03.959224 17476 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:03.959374 17476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:03.959692 17476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:03.962785 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "d61771aac952491d8473090ebd0863c1"
format_stamp: "Formatted at 2026-09-24 19:13:03 on dist-test-slave-0lgr"
server_key: "a37f9718bad9279863dab045d39d129f"
server_key_iv: "2f1d7941b5fc7a23edcca37e972df53b"
server_key_version: "encryptionkey@0"
I20260924 19:13:03.963594 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "d61771aac952491d8473090ebd0863c1"
format_stamp: "Formatted at 2026-09-24 19:13:03 on dist-test-slave-0lgr"
server_key: "a37f9718bad9279863dab045d39d129f"
server_key_iv: "2f1d7941b5fc7a23edcca37e972df53b"
server_key_version: "encryptionkey@0"
I20260924 19:13:03.968971 17476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:13:03.972870 17491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:03.974622 17476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:13:03.974790 17476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "d61771aac952491d8473090ebd0863c1"
format_stamp: "Formatted at 2026-09-24 19:13:03 on dist-test-slave-0lgr"
server_key: "a37f9718bad9279863dab045d39d129f"
server_key_iv: "2f1d7941b5fc7a23edcca37e972df53b"
server_key_version: "encryptionkey@0"
I20260924 19:13:03.974977 17476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:04.020776 17476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:04.025609 17476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:04.025884 17476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:04.045037 17476 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:34731
I20260924 19:13:04.045078 17542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:34731 every 8 connection(s)
I20260924 19:13:04.046552 17476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:13:04.050562 17543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:04.054986 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 17476
I20260924 19:13:04.055214 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:13:04.056097 11337 external_mini_cluster.cc:1428] Setting key 8955bd3290f30db249f09a6ff9b738b5
I20260924 19:13:04.065682 17543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: Bootstrap starting.
I20260924 19:13:04.070303 17543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:04.071780 17543 log.cc:824] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:13:04 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:04.075388 17543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: No bootstrap required, opened a new log
I20260924 19:13:04.081132 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:04.058581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.11.18.126:34731)
0924 19:13:04.059414 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:04.059426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:04.059466 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:13:04.061085 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:04.061107 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:04.061583 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:04.061926 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:04.063729 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:04.065021 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:04.067041 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:04.067777 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:04.067916 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:04.073517 (+ 5601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:04.073526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:04.073542 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:04.073582 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:04.076777 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:04.077601 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:04.077608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:04.077614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:04.077787 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:04.078229 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:04.078234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:04.078239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:04.078836 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:04.079174 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:04.079620 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 19:13:04.080087 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":407,"threads_started":1}
I20260924 19:13:04.085376 17543 raft_consensus.cc:348] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } }
I20260924 19:13:04.085824 17543 raft_consensus.cc:374] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:04.085942 17543 raft_consensus.cc:729] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61771aac952491d8473090ebd0863c1, State: Initialized, Role: FOLLOWER
I20260924 19:13:04.086609 17543 consensus_queue.cc:260] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } }
I20260924 19:13:04.086875 17543 raft_consensus.cc:388] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:13:04.087033 17543 raft_consensus.cc:482] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:13:04.087234 17543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:04.090349 17543 raft_consensus.cc:504] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } }
I20260924 19:13:04.090991 17543 leader_election.cc:302] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d61771aac952491d8473090ebd0863c1; no voters:
I20260924 19:13:04.091601 17543 leader_election.cc:288] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:13:04.091740 17548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:04.092439 17548 raft_consensus.cc:686] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 1 LEADER]: Becoming Leader. State: Replica: d61771aac952491d8473090ebd0863c1, State: Running, Role: LEADER
I20260924 19:13:04.093066 17548 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } }
I20260924 19:13:04.093775 17543 sys_catalog.cc:564] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:04.096944 17549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } }
I20260924 19:13:04.097405 17549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:04.097745 17550 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61771aac952491d8473090ebd0863c1. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } }
I20260924 19:13:04.098214 17550 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:04.104558 17558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:13:04.111039 17558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:13:04.119603 17558 catalog_manager.cc:1347] Generated new cluster ID: c345268d6d3a479aa0c4616a08853079
I20260924 19:13:04.119732 17558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:13:04.151262 17558 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:13:04.152853 17558 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:13:04.165567 17558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: Generated new TSK 0
I20260924 19:13:04.166574 17558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab.
Entry for principal oryx/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab.
I20260924 19:13:04.222921 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34731
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab
--principal=oryx/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:04.565147 17569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:04.565623 17569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:04.565788 17569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:04.573207 17569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:04.573421 17569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:04.573652 17569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:04.584204 17569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab
--principal=oryx/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:04.585755 17569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:04.587080 17569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:04.601953 17575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:04.602041 17574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:04.602010 17577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:04.604538 17569 server_base.cc:1034] running on GCE node
I20260924 19:13:04.605412 17569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:04.606547 17569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:04.608063 17569 hybrid_clock.cc:648] HybridClock initialized: now 1790277184607993 us; error 62 us; skew 500 ppm
I20260924 19:13:04.608577 17569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:04 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277184, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:04.612411 17569 init.cc:377] Logged in from keytab as oryx/127.11.18.65@KRBTEST.COM (short username oryx)
I20260924 19:13:04.614277 17569 webserver.cc:460] Webserver started at http://127.11.18.65:41973/ using document root <none> and password file <none>
I20260924 19:13:04.615061 17569 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:04.615247 17569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:04.615593 17569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:04.618966 17569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "8264c37ea70a460e94c0be82040316c4"
format_stamp: "Formatted at 2026-09-24 19:13:04 on dist-test-slave-0lgr"
server_key: "c81e13fbfde1a988189c1ab88f7c73e9"
server_key_iv: "0652ab2814e6d274d4720d88c2d8b314"
server_key_version: "encryptionkey@0"
I20260924 19:13:04.619956 17569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "8264c37ea70a460e94c0be82040316c4"
format_stamp: "Formatted at 2026-09-24 19:13:04 on dist-test-slave-0lgr"
server_key: "c81e13fbfde1a988189c1ab88f7c73e9"
server_key_iv: "0652ab2814e6d274d4720d88c2d8b314"
server_key_version: "encryptionkey@0"
I20260924 19:13:04.626631 17569 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 19:13:04.630286 17584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:04.631810 17569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:04.632017 17569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "8264c37ea70a460e94c0be82040316c4"
format_stamp: "Formatted at 2026-09-24 19:13:04 on dist-test-slave-0lgr"
server_key: "c81e13fbfde1a988189c1ab88f7c73e9"
server_key_iv: "0652ab2814e6d274d4720d88c2d8b314"
server_key_version: "encryptionkey@0"
I20260924 19:13:04.632236 17569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:04.674350 17569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:04.679867 17569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:04.680302 17569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:04.681453 17569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:04.683158 17569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:04.683240 17569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:04.683386 17569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:04.683461 17569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:04.721563 17569 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:37241
I20260924 19:13:04.721695 17696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:37241 every 8 connection(s)
I20260924 19:13:04.722990 17569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:04.727789 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 17569
I20260924 19:13:04.727985 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:13:04.728652 11337 external_mini_cluster.cc:1428] Setting key e23439d1d7cb83a232b63092a55659c3
Sep 24 19:13:04 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277184, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:04.745260 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:04.725887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:60019 (local address 127.11.18.126:34731)
0924 19:13:04.726399 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:04.726405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:04.727233 (+ 828us) server_negotiation.cc:407] Connection header received
0924 19:13:04.729199 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:04.729205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:04.729298 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:04.729468 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:04.731154 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:04.731883 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:04.733540 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:04.733939 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:04.734081 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:04.738212 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:04.738217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:04.738221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:04.738274 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:04.741288 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:04.742384 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:04.742389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:04.742392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:04.742526 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:04.743100 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:04.743104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:04.743107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:04.743476 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:04.743723 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:04.744714 (+ 991us) server_negotiation.cc:299] Negotiation successful
0924 19:13:04.744966 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":262,"threads_started":1}
I20260924 19:13:04.746843 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:04.726392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:60019)
0924 19:13:04.727077 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:04.727114 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:04.728931 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:04.729752 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:04.729764 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:04.730430 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:04.730988 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:04.731004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:04.732051 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:04.732061 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:04.733379 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:04.733406 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:04.734168 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:04.734177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:04.734432 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:04.735152 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:04.735175 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:04.737950 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:04.741530 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:04.741544 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:04.741567 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:04.742232 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:04.742681 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:04.742689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:04.742695 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:04.742951 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:04.743700 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:04.743712 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:04.744198 (+ 486us) client_negotiation.cc:769] Sending connection context
0924 19:13:04.744542 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 19:13:04.745168 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":266,"threads_started":1}
I20260924 19:13:04.748740 17697 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34731
WARNING: no policy specified for oryx/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:13:04.749109 17697 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:04.750029 17697 heartbeater.cc:511] Master 127.11.18.126:34731 requested a full tablet report, sending...
I20260924 19:13:04.752383 17508 ts_manager.cc:194] Registered new tserver with Master: 8264c37ea70a460e94c0be82040316c4 (127.11.18.65:37241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.66@KRBTEST.COM" created.
I20260924 19:13:04.754643 17508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:60019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab.
Entry for principal oryx/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab.
I20260924 19:13:04.782706 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34731
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab
--principal=oryx/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:05.120692 17703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:05.121044 17703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:05.121224 17703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:05.128683 17703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:05.128867 17703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:05.129055 17703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:05.139591 17703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab
--principal=oryx/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:05.140993 17703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:05.142383 17703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:05.156205 17708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:05.156345 17711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:05.156231 17709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:05.157019 17703 server_base.cc:1034] running on GCE node
I20260924 19:13:05.157766 17703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:05.158790 17703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:05.160046 17703 hybrid_clock.cc:648] HybridClock initialized: now 1790277185159985 us; error 59 us; skew 500 ppm
I20260924 19:13:05.160425 17703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:05 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277185, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:05.170121 17703 init.cc:377] Logged in from keytab as oryx/127.11.18.66@KRBTEST.COM (short username oryx)
I20260924 19:13:05.171710 17703 webserver.cc:460] Webserver started at http://127.11.18.66:34165/ using document root <none> and password file <none>
I20260924 19:13:05.172371 17703 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:05.172478 17703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:05.172721 17703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:05.175660 17703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "bb0d01f02d064ea0bbb746e824d86c52"
format_stamp: "Formatted at 2026-09-24 19:13:05 on dist-test-slave-0lgr"
server_key: "f84c2f22879177e773a3b5e266a25e9e"
server_key_iv: "dd2ed8f0b08ccca516ff9bede40b496a"
server_key_version: "encryptionkey@0"
I20260924 19:13:05.176396 17703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "bb0d01f02d064ea0bbb746e824d86c52"
format_stamp: "Formatted at 2026-09-24 19:13:05 on dist-test-slave-0lgr"
server_key: "f84c2f22879177e773a3b5e266a25e9e"
server_key_iv: "dd2ed8f0b08ccca516ff9bede40b496a"
server_key_version: "encryptionkey@0"
I20260924 19:13:05.181999 17703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:13:05.185675 17718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:05.187192 17703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:05.187366 17703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "bb0d01f02d064ea0bbb746e824d86c52"
format_stamp: "Formatted at 2026-09-24 19:13:05 on dist-test-slave-0lgr"
server_key: "f84c2f22879177e773a3b5e266a25e9e"
server_key_iv: "dd2ed8f0b08ccca516ff9bede40b496a"
server_key_version: "encryptionkey@0"
I20260924 19:13:05.187590 17703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:05.199373 17703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:05.203691 17703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:05.203980 17703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:05.204957 17703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:05.206430 17703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:05.206502 17703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:05.206622 17703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:05.206681 17703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:05.242347 17703 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:40157
I20260924 19:13:05.242401 17830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:40157 every 8 connection(s)
I20260924 19:13:05.243772 17703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:05.245312 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 17703
I20260924 19:13:05.245568 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:13:05.246093 11337 external_mini_cluster.cc:1428] Setting key d2660508adbb5dcd59899fc84c8874b4
Sep 24 19:13:05 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277185, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:13:05.267012 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:05.246761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:54231 (local address 127.11.18.126:34731)
0924 19:13:05.247417 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:05.247424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:05.248306 (+ 882us) server_negotiation.cc:407] Connection header received
0924 19:13:05.250495 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:05.250501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:05.250604 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:05.250749 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:05.252451 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:05.253238 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:05.255030 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:05.255528 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:05.255689 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:05.260211 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:05.260218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:05.260222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:05.260262 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:05.263332 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:05.264251 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:05.264265 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:05.264268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:05.264409 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:05.264908 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:05.264916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:05.264919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:05.265288 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:05.265521 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:05.266314 (+ 793us) server_negotiation.cc:299] Negotiation successful
0924 19:13:05.266587 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":412,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.67@KRBTEST.COM" created.
I20260924 19:13:05.268375 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:05.247248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:54231)
0924 19:13:05.248118 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:05.248187 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:05.250180 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:05.251018 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:05.251027 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:05.251590 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:05.252238 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:05.252258 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:05.253452 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:05.253463 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:05.254842 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:05.254855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:05.255788 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:05.255798 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:05.256076 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:05.256888 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:05.256915 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:05.259856 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:05.263540 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:05.263551 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:05.263574 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:05.264089 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:05.264548 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:05.264555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:05.264559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:05.264778 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:05.265507 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:05.265516 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:05.265914 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 19:13:05.266230 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 19:13:05.266855 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":350,"threads_started":1}
I20260924 19:13:05.270277 17831 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34731
I20260924 19:13:05.270642 17831 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:05.271445 17831 heartbeater.cc:511] Master 127.11.18.126:34731 requested a full tablet report, sending...
I20260924 19:13:05.273078 17508 ts_manager.cc:194] Registered new tserver with Master: bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157)
I20260924 19:13:05.274398 17508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:54231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab.
Entry for principal oryx/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab.
I20260924 19:13:05.297132 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34731
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab
--principal=oryx/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:05.628634 17837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:05.628940 17837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:05.629093 17837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:05.636546 17837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:05.636656 17837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:05.636831 17837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:05.647226 17837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab
--principal=oryx/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:05.648515 17837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:05.649821 17837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:05.664275 17842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:05.664299 17843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:05.664212 17845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:05.664978 17837 server_base.cc:1034] running on GCE node
I20260924 19:13:05.665721 17837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:05.666707 17837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:05.667991 17837 hybrid_clock.cc:648] HybridClock initialized: now 1790277185667886 us; error 75 us; skew 500 ppm
I20260924 19:13:05.668350 17837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:05 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277185, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:05.671629 17837 init.cc:377] Logged in from keytab as oryx/127.11.18.67@KRBTEST.COM (short username oryx)
I20260924 19:13:05.673110 17837 webserver.cc:460] Webserver started at http://127.11.18.67:38975/ using document root <none> and password file <none>
I20260924 19:13:05.673867 17837 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:05.674024 17837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:05.674326 17837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:05.677270 17837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "80c8923bc3f947d79e290801c4f0ee8c"
format_stamp: "Formatted at 2026-09-24 19:13:05 on dist-test-slave-0lgr"
server_key: "f4cd3bb1ef3f3b4800308ba6eae4d3a1"
server_key_iv: "daa3fe364911766cf82c0e0fe8725431"
server_key_version: "encryptionkey@0"
I20260924 19:13:05.678311 17837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "80c8923bc3f947d79e290801c4f0ee8c"
format_stamp: "Formatted at 2026-09-24 19:13:05 on dist-test-slave-0lgr"
server_key: "f4cd3bb1ef3f3b4800308ba6eae4d3a1"
server_key_iv: "daa3fe364911766cf82c0e0fe8725431"
server_key_version: "encryptionkey@0"
I20260924 19:13:05.683902 17837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:13:05.687803 17852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:05.689203 17837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:13:05.689420 17837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "80c8923bc3f947d79e290801c4f0ee8c"
format_stamp: "Formatted at 2026-09-24 19:13:05 on dist-test-slave-0lgr"
server_key: "f4cd3bb1ef3f3b4800308ba6eae4d3a1"
server_key_iv: "daa3fe364911766cf82c0e0fe8725431"
server_key_version: "encryptionkey@0"
I20260924 19:13:05.689627 17837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:05.718973 17837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:05.732978 17837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:05.733399 17837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:05.734829 17837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:05.736910 17837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:05.737021 17837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:05.737154 17837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:05.737303 17837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 19:13:05.758972 17697 heartbeater.cc:503] Master 127.11.18.126:34731 was elected leader, sending a full tablet report...
I20260924 19:13:05.774677 17837 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:37589
I20260924 19:13:05.774742 17964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:37589 every 8 connection(s)
I20260924 19:13:05.776048 17837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:05.779870 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 17837
I20260924 19:13:05.780069 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:13:05.780660 11337 external_mini_cluster.cc:1428] Setting key dee7119bc51511622a1aa18cc0cef98b
Sep 24 19:13:05 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277185, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:05.798890 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:05.778916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:48459 (local address 127.11.18.126:34731)
0924 19:13:05.779515 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:05.779520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:05.780396 (+ 876us) server_negotiation.cc:407] Connection header received
0924 19:13:05.782900 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:05.782906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:05.782976 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:05.783108 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:05.784751 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:05.785603 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:05.787204 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:05.787718 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:05.787849 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:05.791452 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:05.791459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:05.791464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:05.791508 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:05.794999 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:05.795964 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:05.795972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:05.795978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:05.796164 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:05.796663 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:05.796670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:05.796674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:05.796973 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:05.797160 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:05.798101 (+ 941us) server_negotiation.cc:299] Negotiation successful
0924 19:13:05.798437 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":271,"threads_started":1}
I20260924 19:13:05.799865 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:05.779260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:48459)
0924 19:13:05.780113 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:05.780162 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:05.782567 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:05.783349 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:05.783357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:05.784027 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:05.784594 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:05.784610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:05.785753 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:05.785757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:05.787058 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:05.787071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:05.787935 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:05.787940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:05.788159 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:05.788837 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:05.788866 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:05.791150 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:05.795196 (+ 4046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:05.795224 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:05.795238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:05.795812 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:05.796306 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:05.796312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:05.796316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:05.796535 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:05.797123 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:05.797132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:05.797686 (+ 554us) client_negotiation.cc:769] Sending connection context
0924 19:13:05.798006 (+ 320us) client_negotiation.cc:240] Negotiation successful
0924 19:13:05.798633 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":334,"threads_started":1}
I20260924 19:13:05.801436 17965 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34731
I20260924 19:13:05.801759 17965 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:05.802562 17965 heartbeater.cc:511] Master 127.11.18.126:34731 requested a full tablet report, sending...
I20260924 19:13:05.804047 17508 ts_manager.cc:194] Registered new tserver with Master: 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589)
I20260924 19:13:05.804806 17508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:48459
I20260924 19:13:05.809207 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.18.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.125.keytab.
Entry for principal oryx/127.11.18.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.125.keytab.
I20260924 19:13:05.858167 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.125:35333
--webserver_interface=127.11.18.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.125.keytab
--principal=oryx/127.11.18.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:34731,127.11.18.125:35333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:06.208983 17971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:06.209306 17971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:06.209513 17971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:06.217137 17971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:06.217260 17971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:06.217306 17971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:06.217447 17971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:06.217489 17971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:06.227993 17971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:34731,127.11.18.125:35333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.125.keytab
--principal=oryx/127.11.18.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.125:35333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.18.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
W20260924 19:13:06.228346 17971 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.18.126:34731,127.11.18.125:35333'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 19:13:06.229426 17971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:06.230683 17971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:06.244626 17977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:06.244975 17979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:06.245860 17976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:06.247651 17971 server_base.cc:1034] running on GCE node
I20260924 19:13:06.248843 17971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:06.250883 17971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:06.252175 17971 hybrid_clock.cc:648] HybridClock initialized: now 1790277186252087 us; error 60 us; skew 500 ppm
I20260924 19:13:06.252599 17971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:06 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277186, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:06.256667 17971 init.cc:377] Logged in from keytab as oryx/127.11.18.125@KRBTEST.COM (short username oryx)
I20260924 19:13:06.258598 17971 webserver.cc:460] Webserver started at http://127.11.18.125:38465/ using document root <none> and password file <none>
I20260924 19:13:06.259363 17971 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:06.259501 17971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:06.259829 17971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:06.263085 17971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data/instance:
uuid: "ddd1d7a338bb4df4b3777224dafcfd12"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "49e0d47227f3d643fbb48feb193a79c5"
server_key_iv: "b0f12b8f82ac2a5de8ce72a1a6f4fbcf"
server_key_version: "encryptionkey@0"
I20260924 19:13:06.263890 17971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal/instance:
uuid: "ddd1d7a338bb4df4b3777224dafcfd12"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "49e0d47227f3d643fbb48feb193a79c5"
server_key_iv: "b0f12b8f82ac2a5de8ce72a1a6f4fbcf"
server_key_version: "encryptionkey@0"
I20260924 19:13:06.269698 17971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:13:06.273842 17986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:06.275691 17971 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:13:06.275929 17971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
uuid: "ddd1d7a338bb4df4b3777224dafcfd12"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "49e0d47227f3d643fbb48feb193a79c5"
server_key_iv: "b0f12b8f82ac2a5de8ce72a1a6f4fbcf"
server_key_version: "encryptionkey@0"
I20260924 19:13:06.276199 17971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:06.277880 17831 heartbeater.cc:503] Master 127.11.18.126:34731 was elected leader, sending a full tablet report...
I20260924 19:13:06.291395 17971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:06.297211 17971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:06.297662 17971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:06.317665 17971 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.125:35333
I20260924 19:13:06.317687 18037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.125:35333 every 8 connection(s)
I20260924 19:13:06.319121 17971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
I20260924 19:13:06.320539 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 17971
I20260924 19:13:06.324338 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:06.331998 18038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:13:06 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.125@KRBTEST.COM
Sep 24 19:13:06 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277186, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.125@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:06.346375 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.323095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.11.18.125:35333)
0924 19:13:06.323996 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:06.324011 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:06.324052 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:13:06.325917 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:06.325940 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:06.326340 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:06.326781 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:06.328342 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.329778 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.331067 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.331755 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.331965 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.336997 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:06.337006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:06.337020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:06.337090 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:06.341096 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.342099 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.342107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.342114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.342357 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.342922 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.342931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.342936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.343779 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:06.344116 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:06.344541 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 19:13:06.345021 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":372,"threads_started":1}
I20260924 19:13:06.352797 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.334984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.11.18.126:34731)
0924 19:13:06.335480 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:06.335487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:06.336292 (+ 805us) server_negotiation.cc:407] Connection header received
0924 19:13:06.336494 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:06.336502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:06.336603 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:06.336723 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:06.338555 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.339493 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.341629 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.342238 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.342358 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.346273 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:06.346278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:06.346281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:06.346359 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:06.349799 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.350645 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.350650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.350652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.350774 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.351255 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.351259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.351261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.351569 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:06.351723 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:06.352364 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 19:13:06.352571 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260924 19:13:06.352807 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.334972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:47924)
0924 19:13:06.336139 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:06.336182 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:06.336347 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:06.336866 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:06.336874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:06.337944 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:06.338367 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.338385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.339652 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.339657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.341489 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.341500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.342461 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.342469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.342677 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.343281 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:06.343302 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:06.346061 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:06.349991 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.350001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.350008 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.350488 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.350912 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.350916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.350927 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.351111 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.351752 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:06.351758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:06.352005 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 19:13:06.352235 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 19:13:06.352519 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":633,"threads_started":1}
I20260924 19:13:06.354969 18038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:13:06 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277186, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.125@KRBTEST.COM for oryx/127.11.18.125@KRBTEST.COM
I20260924 19:13:06.368886 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.356062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.0.0.1:52196)
0924 19:13:06.356257 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:06.356278 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:06.356401 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:06.356706 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:06.356709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:06.357059 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:06.357187 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.357194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.358372 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.358377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.359371 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.359383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.360113 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.360118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.360250 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.360880 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:06.360898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:06.363185 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:06.366491 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.366495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.366498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.366903 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.367300 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.367304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.367306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.367507 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.368061 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:06.368065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:06.368203 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 19:13:06.368326 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 19:13:06.368600 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:13:06.369448 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.355989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.11.18.125:35333)
0924 19:13:06.356167 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:06.356171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:06.356375 (+ 204us) server_negotiation.cc:407] Connection header received
0924 19:13:06.356518 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:06.356523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:06.356584 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:06.356682 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:06.357310 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.358188 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.359526 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.359960 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.360061 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.363366 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:06.363371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:06.363374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:06.363405 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:06.366274 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.367037 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.367041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.367044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.367186 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.367616 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.367620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.367622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.367911 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:06.368038 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:06.368421 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 19:13:06.368825 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":67}
I20260924 19:13:06.374501 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Bootstrap starting.
I20260924 19:13:06.377556 18038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:06.378654 18038 log.cc:824] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:06.381847 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: No bootstrap required, opened a new log
I20260924 19:13:06.389026 18038 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } }
I20260924 19:13:06.389642 18038 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:06.389722 18038 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd1d7a338bb4df4b3777224dafcfd12, State: Initialized, Role: FOLLOWER
I20260924 19:13:06.390321 18038 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } }
I20260924 19:13:06.391202 18045 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } } }
I20260924 19:13:06.391571 18045 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:13:06.392235 18038 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:06.406360 17971 master_runner.cc:428] Detected that this master ddd1d7a338bb4df4b3777224dafcfd12 is joining an existing cluster
I20260924 19:13:06.406517 17971 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.18.125:35333
W20260924 19:13:06.411957 18056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:13:06.412112 18056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
WARNING: no policy specified for oryx/127.11.18.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.124@KRBTEST.COM" created.
I20260924 19:13:06.429455 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.411839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.11.18.125:35333)
0924 19:13:06.412045 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:06.412051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:06.413794 (+ 1743us) server_negotiation.cc:407] Connection header received
0924 19:13:06.414016 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:06.414023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:06.414089 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:06.414152 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:06.415491 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.416497 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.418436 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.419143 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.419224 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.422071 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:06.422077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:06.422081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:06.422118 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:06.426090 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.427067 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.427071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.427073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.427205 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.427619 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.427624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.427627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.428147 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:06.428332 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:06.428668 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 19:13:06.429006 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:13:06.430235 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.412033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.11.18.126:34731)
0924 19:13:06.412218 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:06.412225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:06.412864 (+ 639us) server_negotiation.cc:407] Connection header received
0924 19:13:06.413098 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:06.413108 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:06.413173 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:06.413297 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:06.414957 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.416123 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.418331 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.419011 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.419122 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.422130 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:06.422136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:06.422140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:06.422186 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:06.426111 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.427756 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.427762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.427765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.427918 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.428412 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.428417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.428420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.428825 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:06.428966 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:06.429652 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 19:13:06.429892 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:13:06.430438 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.412988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.0.0.1:52200)
0924 19:13:06.413703 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:06.413726 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:06.413884 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:06.414320 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:06.414327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:06.415030 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:06.415335 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.415346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.416701 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.416706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.418306 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.418322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.419527 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.419532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.419677 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.420403 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:06.420433 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:06.421870 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:06.426247 (+ 4377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.426252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.426257 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.426921 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.427321 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.427332 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.427337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.427499 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.428253 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:06.428261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:06.428418 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 19:13:06.429723 (+ 1305us) client_negotiation.cc:240] Negotiation successful
0924 19:13:06.430010 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":30,"thread_start_us":471,"threads_started":1}
I20260924 19:13:06.431319 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.411384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:47936)
0924 19:13:06.412277 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:06.412333 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:06.412949 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:06.413540 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:06.413545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:06.414288 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:06.414755 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.414769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.416292 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.416297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.418129 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.418144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.419429 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.419436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.419619 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.420520 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:06.420542 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:06.421932 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:06.427076 (+ 5144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.427081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.427085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.427568 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.428092 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.428098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.428101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.428242 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.429019 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:06.429023 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:06.429213 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 19:13:06.430527 (+ 1314us) client_negotiation.cc:240] Negotiation successful
0924 19:13:06.430858 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":2,"spinlock_wait_cycles":29056,"thread_start_us":1795,"threads_started":1}
Sep 24 19:13:06 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.126@KRBTEST.COM for oryx/127.11.18.125@KRBTEST.COM
I20260924 19:13:06.460222 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.437092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.11.18.125:35333)
0924 19:13:06.437279 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:06.437286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:06.442200 (+ 4914us) server_negotiation.cc:407] Connection header received
0924 19:13:06.442624 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:06.442630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:06.442722 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:06.442838 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:06.444405 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.445649 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.447770 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.448509 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.448665 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.453312 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:06.453319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:06.453358 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:06.453402 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:06.456745 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.457803 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.457807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.457810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.457938 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.458557 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.458562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.458564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.458950 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:06.459168 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:06.459661 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 19:13:06.459947 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:13:06.460352 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.437090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.0.0.1:52206)
0924 19:13:06.442035 (+ 4945us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:06.442090 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:06.442366 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:06.442911 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:06.442921 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:06.443873 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:06.444204 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.444218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.445800 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.445808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.447586 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.447606 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.448718 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.448727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.448986 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.449953 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:06.449979 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:06.453073 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:06.456913 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.456929 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.456937 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.457683 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.458096 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.458101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.458105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.458300 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.459117 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:06.459126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:06.459334 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 19:13:06.459557 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 19:13:06.459902 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4606,"thread_start_us":4441,"threads_started":1}
I20260924 19:13:06.462509 17508 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.18.125:35333: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12"
member_type: NON_VOTER
last_known_addr {
host: "127.11.18.125"
port: 35333
}
attrs {
promote: true
}
}
dest_uuid: "d61771aac952491d8473090ebd0863c1"
cas_config_opid_index: -1
I20260924 19:13:06.463624 17508 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.124.keytab.
Entry for principal oryx/127.11.18.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.124.keytab.
I20260924 19:13:06.467104 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.124:37469
--webserver_interface=127.11.18.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.124.keytab
--principal=oryx/127.11.18.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:34731,127.11.18.125:35333,127.11.18.124:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:13:06.475040 18068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ddd1d7a338bb4df4b3777224dafcfd12 (127.11.18.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } }
I20260924 19:13:06.476399 18013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:06.479662 18068 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.18.125:35333
I20260924 19:13:06.479851 18013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Refusing update from remote peer d61771aac952491d8473090ebd0863c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 19:13:06.480127 18069 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } }
I20260924 19:13:06.480520 18069 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:06.480558 17971 master.cc:537] Master@127.11.18.125:35333 shutting down...
I20260924 19:13:06.481077 18068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:13:06.485251 17494 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 19:13:06.498688 17971 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:06.499318 17971 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:06.499581 17971 tablet_replica.cc:330] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: stopping tablet replica
W20260924 19:13:06.504091 17494 consensus_peers.cc:487] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 -> Peer ddd1d7a338bb4df4b3777224dafcfd12 (127.11.18.125:35333): Couldn't send request to peer ddd1d7a338bb4df4b3777224dafcfd12. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:13:06.516680 17971 master.cc:552] Master@127.11.18.125:35333 shutdown complete.
I20260924 19:13:06.516798 17971 master_runner.cc:305] Clearing existing system tablet
I20260924 19:13:06.517177 17971 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:06.517294 17971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:06.519580 17971 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 19:13:06.521013 18074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:06.521468 17971 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:06.521723 17971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:13:06.521809 17971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
uuid: "ddd1d7a338bb4df4b3777224dafcfd12"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "49e0d47227f3d643fbb48feb193a79c5"
server_key_iv: "b0f12b8f82ac2a5de8ce72a1a6f4fbcf"
server_key_version: "encryptionkey@0"
I20260924 19:13:06.525120 17971 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:06.530154 17971 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:13:06.530362 17971 log.cc:1196] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 19:13:06.530752 17971 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Deleting consensus metadata
I20260924 19:13:06.531013 17971 master_runner.cc:315] Copying system tablet from 127.11.18.126:34731
I20260924 19:13:06.533841 17971 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: tablet copy: Beginning tablet copy session from remote peer at address 127.11.18.126:34731
I20260924 19:13:06.547775 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.534623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.11.18.126:34731)
0924 19:13:06.534816 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:06.534823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:06.535264 (+ 441us) server_negotiation.cc:407] Connection header received
0924 19:13:06.535606 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:06.535612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:06.535693 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:06.535825 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:06.536799 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.537798 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.539112 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.539704 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.539952 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.542130 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:06.542138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:06.542154 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:06.542187 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:06.544931 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.545737 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.545743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.545747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.545900 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:06.546291 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:06.546296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:06.546299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:06.546587 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:06.546749 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:06.547157 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 19:13:06.547382 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 19:13:06.548139 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:06.534620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:47950)
0924 19:13:06.535136 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:06.535164 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:06.535343 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:06.535855 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:06.535860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:06.536336 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:06.536643 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.536653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.537941 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.537946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.538904 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:06.538916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:06.539972 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:06.539979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:06.540177 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:06.541024 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:06.541041 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:06.541917 (+ 876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:06.545160 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.545163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.545166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.545610 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.546033 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:06.546037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:06.546040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:06.546166 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:06.546723 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:06.546728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:06.546896 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 19:13:06.547050 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 19:13:06.547390 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":277,"threads_started":1}
I20260924 19:13:06.548983 17528 tablet_copy_service.cc:140] P d61771aac952491d8473090ebd0863c1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ddd1d7a338bb4df4b3777224dafcfd12 ({username='oryx', principal='oryx/127.11.18.125@KRBTEST.COM'} at 127.0.0.1:47950)
I20260924 19:13:06.549252 17528 tablet_copy_service.cc:161] P d61771aac952491d8473090ebd0863c1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ddd1d7a338bb4df4b3777224dafcfd12 at {username='oryx', principal='oryx/127.11.18.125@KRBTEST.COM'} at 127.0.0.1:47950: session id = ddd1d7a338bb4df4b3777224dafcfd12-00000000000000000000000000000000
I20260924 19:13:06.552395 17528 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:13:06.554358 17971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:13:06.559262 17971 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: tablet copy: Starting download of 0 data blocks...
I20260924 19:13:06.559602 17971 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: tablet copy: Starting download of 1 WAL segments...
I20260924 19:13:06.562268 17971 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:13:06.565689 17528 tablet_copy_service.cc:342] P d61771aac952491d8473090ebd0863c1: Request end of tablet copy session ddd1d7a338bb4df4b3777224dafcfd12-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.18.125@KRBTEST.COM'} at 127.0.0.1:47950
I20260924 19:13:06.565913 17528 tablet_copy_service.cc:433] P d61771aac952491d8473090ebd0863c1: ending tablet copy session ddd1d7a338bb4df4b3777224dafcfd12-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ddd1d7a338bb4df4b3777224dafcfd12
I20260924 19:13:06.576884 17971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:06.580330 18081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:06.589767 18082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:06.590488 18084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:06.591020 17971 server_base.cc:1034] running on GCE node
I20260924 19:13:06.591833 17971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:06.592578 17971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:06.593814 17971 hybrid_clock.cc:648] HybridClock initialized: now 1790277186593721 us; error 69 us; skew 500 ppm
I20260924 19:13:06.594110 17971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:06 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277186, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:06.596408 17971 init.cc:377] Logged in from keytab as oryx/127.11.18.125@KRBTEST.COM (short username oryx)
I20260924 19:13:06.603804 17971 webserver.cc:460] Webserver started at http://127.11.18.125:33297/ using document root <none> and password file <none>
I20260924 19:13:06.604187 17971 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:06.604285 17971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:06.607056 17971 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:13:06.608865 18091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:06.609342 17971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260924 19:13:06.609485 17971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
uuid: "ddd1d7a338bb4df4b3777224dafcfd12"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "49e0d47227f3d643fbb48feb193a79c5"
server_key_iv: "b0f12b8f82ac2a5de8ce72a1a6f4fbcf"
server_key_version: "encryptionkey@0"
I20260924 19:13:06.609665 17971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:06.626058 17971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:06.629597 17971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:06.654495 17971 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.125:35333
I20260924 19:13:06.654515 18142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.125:35333 every 8 connection(s)
I20260924 19:13:06.655395 17971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
I20260924 19:13:06.662366 18143 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:13:06.666577 18143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Bootstrap starting.
I20260924 19:13:06.680459 18143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:13:06.680953 18143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Bootstrap complete.
I20260924 19:13:06.681926 18143 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } }
I20260924 19:13:06.682176 18143 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ddd1d7a338bb4df4b3777224dafcfd12, State: Initialized, Role: LEARNER
I20260924 19:13:06.682395 18143 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } }
I20260924 19:13:06.683385 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } }
I20260924 19:13:06.683684 18143 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:06.683791 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:13:06.694103 18157 catalog_manager.cc:1259] Loaded cluster ID: c345268d6d3a479aa0c4616a08853079
I20260924 19:13:06.694244 18157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: loading cluster ID for follower catalog manager: success
I20260924 19:13:06.697454 18157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: acquiring CA information for follower catalog manager: success
I20260924 19:13:06.699635 18157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:13:06.807550 17965 heartbeater.cc:503] Master 127.11.18.126:34731 was elected leader, sending a full tablet report...
W20260924 19:13:06.931493 18070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:06.931806 18070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:06.931945 18070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:06.939806 18070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:06.939942 18070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:06.939988 18070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:06.940104 18070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:06.940196 18070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:06.950722 18070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:34731,127.11.18.125:35333,127.11.18.124:37469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.124.keytab
--principal=oryx/127.11.18.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.124:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.18.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:06.952149 18070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:06.953691 18070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:06.966857 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:06.966928 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:06.966948 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:06.967867 18070 server_base.cc:1034] running on GCE node
I20260924 19:13:06.969285 18070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:06.971280 18070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:06.972596 18070 hybrid_clock.cc:648] HybridClock initialized: now 1790277186972568 us; error 130 us; skew 500 ppm
I20260924 19:13:06.973071 18070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:06 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277186, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:06.978152 18070 init.cc:377] Logged in from keytab as oryx/127.11.18.124@KRBTEST.COM (short username oryx)
I20260924 19:13:06.980094 18070 webserver.cc:460] Webserver started at http://127.11.18.124:38549/ using document root <none> and password file <none>
I20260924 19:13:06.980885 18070 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:06.981045 18070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:06.981441 18070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:06.984989 18070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data/instance:
uuid: "a93be477dd93422ea19b9d8d06c25c69"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "3b42479984cf621d4744e0a3b640c8f2"
server_key_iv: "84b02d5a6dfec1598ef36064176660b8"
server_key_version: "encryptionkey@0"
I20260924 19:13:06.985963 18070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal/instance:
uuid: "a93be477dd93422ea19b9d8d06c25c69"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "3b42479984cf621d4744e0a3b640c8f2"
server_key_iv: "84b02d5a6dfec1598ef36064176660b8"
server_key_version: "encryptionkey@0"
I20260924 19:13:06.991930 18070 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:13:06.996237 18172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:06.997934 18070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 19:13:06.998129 18070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
uuid: "a93be477dd93422ea19b9d8d06c25c69"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "3b42479984cf621d4744e0a3b640c8f2"
server_key_iv: "84b02d5a6dfec1598ef36064176660b8"
server_key_version: "encryptionkey@0"
I20260924 19:13:06.998358 18070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:07.030169 18070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:07.034996 18070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:07.035290 18070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:07.059350 18070 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.124:37469
I20260924 19:13:07.059379 18223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.124:37469 every 8 connection(s)
I20260924 19:13:07.061185 18070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
I20260924 19:13:07.066152 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:07.066898 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 18070
I20260924 19:13:07.076812 18224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.124@KRBTEST.COM
I20260924 19:13:07.100546 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.078927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.11.18.125:35333)
0924 19:13:07.091952 (+ 13025us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.091960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.092227 (+ 267us) server_negotiation.cc:407] Connection header received
0924 19:13:07.092569 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.092578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.092685 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.092805 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:07.093541 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.095416 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.097923 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.099411 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.099593 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.099843 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.100052 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.100195 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12840,"thread_start_us":12707,"threads_started":1}
I20260924 19:13:07.100548 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.078730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.0.0.1:52216)
0924 19:13:07.092104 (+ 13374us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.092121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.092308 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.093015 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.093019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.093042 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:07.093315 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.093376 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.095579 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.095586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.097766 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.097781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.099625 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.099632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.099813 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.099888 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.099984 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.100139 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13252,"thread_start_us":13136,"threads_started":1}
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277186, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.124@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:07.102255 18118 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 19:13:07.103263 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61771aac952491d8473090ebd0863c1. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } }
I20260924 19:13:07.103533 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:13:07.105999 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.069643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.11.18.124:37469)
0924 19:13:07.073502 (+ 3859us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.073520 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.073573 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:13:07.075777 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.075805 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.076300 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.076830 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.078829 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.081549 (+ 2720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.084338 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.086054 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.086313 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.094403 (+ 8090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.094412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.094425 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.094505 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.098354 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.099268 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.099277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.099294 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.099532 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.100946 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.100950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.100952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.101686 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.101970 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.103275 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.103851 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3564,"thread_start_us":3252,"threads_started":1}
I20260924 19:13:07.112315 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.081439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.11.18.126:34731)
0924 19:13:07.090529 (+ 9090us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.090538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.090564 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:13:07.090636 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.090644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.090734 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.090862 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.094195 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.095559 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.097627 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.098425 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.098555 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.102827 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.102834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.102838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.102884 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.109044 (+ 6160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.110021 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.110027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.110029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.110188 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.110642 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.110647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.110649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.110936 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.111108 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.111605 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.111855 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8938,"thread_start_us":8799,"threads_started":1}
I20260924 19:13:07.112342 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.080678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:47952)
0924 19:13:07.090007 (+ 9329us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.090096 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.090324 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.092185 (+ 1861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.092196 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.093176 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.094015 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.094031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.095728 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.095745 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.097470 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.097483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.098846 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.098855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.099062 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.099822 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.099846 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.102597 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.109223 (+ 6626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.109237 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.109244 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.109842 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.110327 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.110332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.110337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.110495 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.111074 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.111082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.111279 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.111489 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.111855 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8927,"thread_start_us":9775,"threads_started":1}
I20260924 19:13:07.114128 18224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277186, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.124@KRBTEST.COM for oryx/127.11.18.125@KRBTEST.COM
I20260924 19:13:07.127599 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.114881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.0.0.1:52226)
0924 19:13:07.115121 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.115140 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.115237 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.115524 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.115527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.115833 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.115964 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.115972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.117115 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.117119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.118125 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.118136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.118864 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.118868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.118994 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.119580 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.119597 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.121715 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.125014 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.125019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.125022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.125592 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.126041 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.126044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.126047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.126187 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.126876 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.126880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.127033 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.127149 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.127368 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 19:13:07.128028 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.114958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.11.18.125:35333)
0924 19:13:07.115138 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.115143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.115210 (+ 67us) server_negotiation.cc:407] Connection header received
0924 19:13:07.115338 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.115342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.115406 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.115496 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.116140 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.116980 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.118290 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.118717 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.118829 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.121921 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.121927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.121930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.121965 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.124832 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.125727 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.125731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.125733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.125864 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.126386 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.126391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.126393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.126713 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.126860 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.127419 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.127613 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:13:07.129153 18224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277186, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.124@KRBTEST.COM for oryx/127.11.18.124@KRBTEST.COM
I20260924 19:13:07.144727 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.130043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.0.0.1:60498)
0924 19:13:07.130448 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.130469 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.130638 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.131002 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.131017 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.131460 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.131599 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.131608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.132988 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.132997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.134062 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.134075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.134856 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.134862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.135553 (+ 691us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.136237 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.136254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.139043 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.142269 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.142273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.142276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.142645 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.143036 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.143040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.143042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.143211 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.143857 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.143861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.144040 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.144165 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.144423 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320}
I20260924 19:13:07.144994 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.130094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.11.18.124:37469)
0924 19:13:07.130440 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.130447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.130569 (+ 122us) server_negotiation.cc:407] Connection header received
0924 19:13:07.130762 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.130767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.130867 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.130984 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.131750 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.132848 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.134225 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.134665 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.134820 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.139244 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.139249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.139253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.139282 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.142104 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.142778 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.142782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.142784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.142896 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.143351 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.143355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.143358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.143702 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.143830 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.144282 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.144622 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":214}
I20260924 19:13:07.150015 18224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Bootstrap starting.
I20260924 19:13:07.153119 18224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:07.154207 18224 log.cc:824] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:07.156867 18224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: No bootstrap required, opened a new log
I20260924 19:13:07.164726 18224 raft_consensus.cc:348] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } }
I20260924 19:13:07.165076 18224 raft_consensus.cc:374] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:07.165222 18224 raft_consensus.cc:729] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93be477dd93422ea19b9d8d06c25c69, State: Initialized, Role: FOLLOWER
I20260924 19:13:07.165791 18224 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } }
I20260924 19:13:07.166610 18234 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } } }
I20260924 19:13:07.166898 18234 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:13:07.167631 18224 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:13:07.177760 18245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:13:07.177930 18245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:13:07.182706 18070 master_runner.cc:428] Detected that this master a93be477dd93422ea19b9d8d06c25c69 is joining an existing cluster
I20260924 19:13:07.182869 18070 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.18.124:37469
I20260924 19:13:07.213384 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 17569
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.65@KRBTEST.COM
I20260924 19:13:07.227933 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.187943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.11.18.124:37469)
0924 19:13:07.188213 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.188219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.205568 (+ 17349us) server_negotiation.cc:407] Connection header received
0924 19:13:07.205751 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.205759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.205829 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.205930 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.207904 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.209173 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.211511 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.212755 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.212846 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.216328 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.216335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.216340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.216376 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.224653 (+ 8277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.225505 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.225511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.225515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.225644 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.226087 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.226094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.226097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.226573 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.226736 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.227261 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.227515 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:13:07.228433 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.187506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.11.18.125:35333)
0924 19:13:07.187725 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.187732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.205063 (+ 17331us) server_negotiation.cc:407] Connection header received
0924 19:13:07.205372 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.205380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.205466 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.205540 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.207275 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.208382 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.211086 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.211779 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.211879 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.218694 (+ 6815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.218704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.218711 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.218769 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.223283 (+ 4514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.225983 (+ 2700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.225989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.225993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.226166 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.226647 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.226652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.226656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.227001 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.227148 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.227785 (+ 637us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.228039 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:13:07.228932 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.188736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.0.0.1:60514)
0924 19:13:07.205453 (+ 16717us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.205478 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.205616 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.206103 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.206108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.207121 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.207731 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.207744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.209440 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.209446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.211365 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.211379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.213172 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.213179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.213513 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.214402 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.214428 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.216138 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.224819 (+ 8681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.224825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.224830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.225376 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.225823 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.225827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.225831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.225978 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.226816 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.226821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.226984 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.228116 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.228412 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16549,"mutex_wait_us":93,"thread_start_us":16408,"threads_started":1}
I20260924 19:13:07.230466 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.188441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.0.0.1:52228)
0924 19:13:07.204945 (+ 16504us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.204965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.205172 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.205750 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.205755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.206676 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.207090 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.207103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.208579 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.208585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.210898 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.210912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.212513 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.212522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.212745 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.213666 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.213694 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.216857 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.223451 (+ 6594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.223461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.223465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.225835 (+ 2370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.226340 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.226344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.226347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.226508 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.227229 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.227238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.227487 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.228678 (+ 1191us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.229833 (+ 1155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15625,"mutex_wait_us":8,"thread_start_us":15395,"threads_started":1}
I20260924 19:13:07.233829 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.187360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.11.18.126:34731)
0924 19:13:07.187541 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.187566 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.203609 (+ 16043us) server_negotiation.cc:407] Connection header received
0924 19:13:07.205833 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.205840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.205969 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.206046 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.207922 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.209124 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.215016 (+ 5892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.215727 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.215867 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.220907 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.220914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.220921 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.220973 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.225203 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.230711 (+ 5508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.230718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.230721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.231195 (+ 474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.231903 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.231910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.231914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.232338 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.232513 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.233211 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.233486 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:13:07.233875 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.188392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:47958)
0924 19:13:07.203406 (+ 15014us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.203440 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.204828 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.206259 (+ 1431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.206265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.207116 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.207734 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.207744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.209317 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.209349 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.214807 (+ 5458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.214823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.217120 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.217128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.217421 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.218105 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.218129 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.220652 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.229214 (+ 8562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.229218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.229222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.229734 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.231389 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.231394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.231398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.231610 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.232606 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.232612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.232860 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.233039 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.233398 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14761,"mutex_wait_us":147,"spinlock_wait_cycles":3968,"thread_start_us":15822,"threads_started":1}
I20260924 19:13:07.238300 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.214587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.11.18.65:37241)
0924 19:13:07.215342 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.215351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.215380 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:13:07.215446 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.215456 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.215736 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.217561 (+ 1825us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.217859 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.219527 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.221549 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.222353 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.228858 (+ 6505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.229407 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.229422 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.229442 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.229482 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.234624 (+ 5142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.235540 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.235554 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.235561 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.235738 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.236304 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.236309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.236313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.236762 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.237194 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.237581 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.237849 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":440,"threads_started":1}
I20260924 19:13:07.239938 17692 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.126@KRBTEST.COM for oryx/127.11.18.124@KRBTEST.COM
I20260924 19:13:07.260727 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.239737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.0.0.1:60524)
0924 19:13:07.240237 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.240256 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.240388 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.241049 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.241058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.241673 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.241896 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.241915 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.246770 (+ 4855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.246776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.248227 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.248246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.249473 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.249480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.249669 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.250458 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.250486 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.253598 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.257757 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.257764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.257769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.258357 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.258814 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.258819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.258824 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.258999 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.259649 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.259655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.259835 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.259999 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.260319 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375}
I20260924 19:13:07.261135 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.239789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.11.18.124:37469)
0924 19:13:07.240020 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.240024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.240337 (+ 313us) server_negotiation.cc:407] Connection header received
0924 19:13:07.240562 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.240570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.240684 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.240796 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.245074 (+ 4278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.246561 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.248420 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.249251 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.249457 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.253840 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.253847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.253853 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.253893 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.257599 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.258498 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.258504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.258508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.258661 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.259123 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.259129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.259134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.259496 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.259632 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.260265 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.260481 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260924 19:13:07.262830 17508 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.18.124:37469: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69"
member_type: NON_VOTER
last_known_addr {
host: "127.11.18.124"
port: 37469
}
attrs {
promote: true
}
}
dest_uuid: "d61771aac952491d8473090ebd0863c1"
cas_config_opid_index: 5
I20260924 19:13:07.263880 17508 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } }
I20260924 19:13:07.265525 18228 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.266305 18199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:07.270617 18199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Refusing update from remote peer d61771aac952491d8473090ebd0863c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 19:13:07.279525 18228 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.18.124:37469
I20260924 19:13:07.280580 18070 master.cc:537] Master@127.11.18.124:37469 shutting down...
I20260924 19:13:07.281713 18118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 LEARNER]: Refusing update from remote peer d61771aac952491d8473090ebd0863c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 19:13:07.290211 18258 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:07.292199 18258 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 19:13:07.293754 18257 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.294274 18257 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:07.295710 18118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
W20260924 19:13:07.301614 18178 tablet.cc:2369] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:13:07.302922 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: NON_VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: true } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.304411 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:13:07.305023 18228 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: attempting to promote NON_VOTER ddd1d7a338bb4df4b3777224dafcfd12 to VOTER
I20260924 19:13:07.306246 18228 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } }
I20260924 19:13:07.307339 18070 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:07.307797 18070 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:07.308023 18070 tablet_replica.cc:330] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: stopping tablet replica
I20260924 19:13:07.312180 18118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 LEARNER]: Refusing update from remote peer d61771aac952491d8473090ebd0863c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 19:13:07.313410 18228 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 19:13:07.316493 18256 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ddd1d7a338bb4df4b3777224dafcfd12 (127.11.18.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.317103 18118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ddd1d7a338bb4df4b3777224dafcfd12 (127.11.18.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.321015 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.321270 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:13:07.321537 18257 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.321798 18257 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:07.322100 18257 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.322315 18257 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:07.325316 18070 master.cc:552] Master@127.11.18.124:37469 shutdown complete.
I20260924 19:13:07.325443 18070 master_runner.cc:305] Clearing existing system tablet
W20260924 19:13:07.325805 17493 connection.cc:698] client connection to 127.11.18.124:37469 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260924 19:13:07.325874 18070 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:07.325987 18070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:07.327333 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.326864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.0.0.1:60536)
0924 19:13:07.327022 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.327137 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 19:13:07.327684 17493 consensus_peers.cc:487] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 -> Peer a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124:37469): Couldn't send request to peer a93be477dd93422ea19b9d8d06c25c69. Status: Network error: Client connection negotiation failed: client connection to 127.11.18.124:37469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:13:07.329106 18070 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:13:07.330837 18263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:07.331319 18070 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:07.331589 18070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 19:13:07.331693 18070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
uuid: "a93be477dd93422ea19b9d8d06c25c69"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "3b42479984cf621d4744e0a3b640c8f2"
server_key_iv: "84b02d5a6dfec1598ef36064176660b8"
server_key_version: "encryptionkey@0"
I20260924 19:13:07.337796 18070 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:07.345768 18070 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:13:07.345968 18070 log.cc:1196] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 19:13:07.346493 18070 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Deleting consensus metadata
I20260924 19:13:07.346793 18070 master_runner.cc:315] Copying system tablet from 127.11.18.126:34731
I20260924 19:13:07.349764 18070 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: tablet copy: Beginning tablet copy session from remote peer at address 127.11.18.126:34731
I20260924 19:13:07.367092 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.350607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.11.18.126:34731)
0924 19:13:07.350837 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.350844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.351418 (+ 574us) server_negotiation.cc:407] Connection header received
0924 19:13:07.351615 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.351622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.351708 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.351828 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.353217 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.354592 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.356907 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.357707 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.357858 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.360552 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.360559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.360565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.360606 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.363998 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.364891 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.364897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.364900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.365061 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.365543 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.365547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.365550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.365879 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.366016 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.366511 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.366723 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 19:13:07.367483 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.350580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:47964)
0924 19:13:07.351276 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.351301 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.351455 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.351868 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.351873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.352689 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.353014 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.353029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.354755 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.354761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.356712 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.356728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.357974 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.357981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.358229 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.359192 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.359216 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.360304 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.364155 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.364161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.364165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.364753 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.365196 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.365201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.365205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.365433 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.366022 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.366027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.366201 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.366388 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.366717 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":391,"threads_started":1}
I20260924 19:13:07.368362 17528 tablet_copy_service.cc:140] P d61771aac952491d8473090ebd0863c1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a93be477dd93422ea19b9d8d06c25c69 ({username='oryx', principal='oryx/127.11.18.124@KRBTEST.COM'} at 127.0.0.1:47964)
I20260924 19:13:07.368535 17528 tablet_copy_service.cc:161] P d61771aac952491d8473090ebd0863c1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a93be477dd93422ea19b9d8d06c25c69 at {username='oryx', principal='oryx/127.11.18.124@KRBTEST.COM'} at 127.0.0.1:47964: session id = a93be477dd93422ea19b9d8d06c25c69-00000000000000000000000000000000
I20260924 19:13:07.372524 17528 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:13:07.374504 18070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:13:07.381482 18070 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: tablet copy: Starting download of 0 data blocks...
I20260924 19:13:07.381781 18070 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: tablet copy: Starting download of 1 WAL segments...
I20260924 19:13:07.384392 18070 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:13:07.389148 17528 tablet_copy_service.cc:342] P d61771aac952491d8473090ebd0863c1: Request end of tablet copy session a93be477dd93422ea19b9d8d06c25c69-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.18.124@KRBTEST.COM'} at 127.0.0.1:47964
I20260924 19:13:07.389297 17528 tablet_copy_service.cc:433] P d61771aac952491d8473090ebd0863c1: ending tablet copy session a93be477dd93422ea19b9d8d06c25c69-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a93be477dd93422ea19b9d8d06c25c69
I20260924 19:13:07.397859 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 17569
I20260924 19:13:07.402067 18070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:07.405203 18270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:07.411746 18271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:07.427385 18273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:07.428174 18070 server_base.cc:1034] running on GCE node
I20260924 19:13:07.428546 18070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:07.428977 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:37241
--local_ip_for_outbound_sockets=127.11.18.65
--tserver_master_addrs=127.11.18.126:34731
--webserver_port=41973
--webserver_interface=127.11.18.65
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab
--principal=oryx/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333,127.11.18.124:37469 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:13:07.429565 18070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:07.430780 18070 hybrid_clock.cc:648] HybridClock initialized: now 1790277187430741 us; error 29 us; skew 500 ppm
I20260924 19:13:07.431033 18070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277187, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:07.433175 18070 init.cc:377] Logged in from keytab as oryx/127.11.18.124@KRBTEST.COM (short username oryx)
I20260924 19:13:07.439471 18070 webserver.cc:460] Webserver started at http://127.11.18.124:38619/ using document root <none> and password file <none>
I20260924 19:13:07.439813 18070 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:07.439924 18070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:07.443114 18070 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:13:07.445086 18281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:07.445544 18070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:13:07.445719 18070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
uuid: "a93be477dd93422ea19b9d8d06c25c69"
format_stamp: "Formatted at 2026-09-24 19:13:06 on dist-test-slave-0lgr"
server_key: "3b42479984cf621d4744e0a3b640c8f2"
server_key_iv: "84b02d5a6dfec1598ef36064176660b8"
server_key_version: "encryptionkey@0"
I20260924 19:13:07.445998 18070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:07.473829 18070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:07.478075 18070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:07.508181 18070 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.124:37469
I20260924 19:13:07.508266 18332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.124:37469 every 8 connection(s)
I20260924 19:13:07.509418 18070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
I20260924 19:13:07.515727 18333 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:13:07.520498 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Bootstrap starting.
I20260924 19:13:07.541797 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:13:07.542485 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: Bootstrap complete.
I20260924 19:13:07.543792 18333 raft_consensus.cc:348] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } }
I20260924 19:13:07.544117 18333 raft_consensus.cc:729] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a93be477dd93422ea19b9d8d06c25c69, State: Initialized, Role: LEARNER
I20260924 19:13:07.544487 18333 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } }
I20260924 19:13:07.545521 18336 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.546000 18333 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:07.546206 18336 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:13:07.558991 18347 catalog_manager.cc:1259] Loaded cluster ID: c345268d6d3a479aa0c4616a08853079
I20260924 19:13:07.559123 18347 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: loading cluster ID for follower catalog manager: success
I20260924 19:13:07.563011 18347 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: acquiring CA information for follower catalog manager: success
I20260924 19:13:07.565819 18347 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:13:07.758287 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.750739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.0.0.1:60544)
0924 19:13:07.751075 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.751098 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.751283 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.751983 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.751989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.752027 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:07.752259 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.752274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.753726 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.753734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.755929 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.755951 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.757297 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.757304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.757600 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.757645 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.757749 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.757911 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260924 19:13:07.758353 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.750913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.11.18.124:37469)
0924 19:13:07.751575 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.751580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.751597 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:07.751656 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.751660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.751798 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.751910 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:07.752433 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.753569 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.756165 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.757069 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.757250 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.757547 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.757812 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.757970 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":404,"threads_started":1}
I20260924 19:13:07.759662 18308 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 19:13:07.760394 18336 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61771aac952491d8473090ebd0863c1. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:07.760705 18336 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: This master's current role is: LEARNER
W20260924 19:13:07.826543 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:07.826839 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:07.826997 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:07.834755 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:07.834869 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:07.835047 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:07.845135 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab
--principal=oryx/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:37241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=41973
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333,127.11.18.124:37469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:07.846730 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:07.848203 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:07.862706 18353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:07.862839 18356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:07.862861 18354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:07.863287 18277 server_base.cc:1034] running on GCE node
I20260924 19:13:07.863934 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:07.864953 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:07.866189 18277 hybrid_clock.cc:648] HybridClock initialized: now 1790277187866144 us; error 47 us; skew 500 ppm
I20260924 19:13:07.866561 18277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277187, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:07.870441 18277 init.cc:377] Logged in from keytab as oryx/127.11.18.65@KRBTEST.COM (short username oryx)
I20260924 19:13:07.872011 18277 webserver.cc:460] Webserver started at http://127.11.18.65:41973/ using document root <none> and password file <none>
I20260924 19:13:07.872711 18277 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:07.872859 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:07.878542 18277 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:13:07.881806 18363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:07.883342 18277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 19:13:07.883533 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "8264c37ea70a460e94c0be82040316c4"
format_stamp: "Formatted at 2026-09-24 19:13:04 on dist-test-slave-0lgr"
server_key: "c81e13fbfde1a988189c1ab88f7c73e9"
server_key_iv: "0652ab2814e6d274d4720d88c2d8b314"
server_key_version: "encryptionkey@0"
I20260924 19:13:07.884599 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:07.901963 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:07.906654 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:07.906960 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:07.907968 18277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:07.909399 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:07.909489 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:07.909598 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:07.909674 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:07.948513 18277 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:37241
I20260924 19:13:07.948616 18475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:37241 every 8 connection(s)
I20260924 19:13:07.950078 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:07.954054 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 18277
I20260924 19:13:07.954480 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 17703
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.66@KRBTEST.COM
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277187, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.124@KRBTEST.COM
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277187, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.125@KRBTEST.COM
Sep 24 19:13:07 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277187, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:07.990306 18483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.956273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.11.18.66:40157)
0924 19:13:07.966285 (+ 10012us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.966309 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.967172 (+ 863us) server_negotiation.cc:407] Connection header received
0924 19:13:07.967266 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.967276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.967563 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.967730 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.970059 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.971221 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.973101 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.974021 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.974123 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.981423 (+ 7300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.981447 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.981465 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.981517 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.986151 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.987096 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.987105 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.987108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.987276 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.987792 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.987800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.987806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.988301 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.988590 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.989556 (+ 966us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.989838 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9815,"thread_start_us":9649,"threads_started":1}
I20260924 19:13:07.992739 17826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:07.994343 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.953192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:46849 (local address 127.11.18.124:37469)
0924 19:13:07.953420 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.953429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.969847 (+ 16418us) server_negotiation.cc:407] Connection header received
0924 19:13:07.971254 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.971260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.971352 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.971486 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.974623 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.975739 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.978616 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.979283 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.979391 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.985553 (+ 6162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.985562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.985568 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.985613 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.989853 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.990960 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.990966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.990969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.991114 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.991507 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.991512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.991516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.992374 (+ 858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.992547 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.993603 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.993937 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:13:07.994346 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.953312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:46915 (local address 127.11.18.125:35333)
0924 19:13:07.965638 (+ 12326us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.965649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.969390 (+ 3741us) server_negotiation.cc:407] Connection header received
0924 19:13:07.971522 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.971528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.971628 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.971755 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.974623 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.975737 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.978616 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.979354 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.979448 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.985698 (+ 6250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.985703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.985706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.985739 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.989711 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.990754 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.990761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.990815 (+ 54us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.990959 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.991507 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.991512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.991516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.991859 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.992006 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.993663 (+ 1657us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.993886 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12130,"thread_start_us":12320,"threads_started":1}
I20260924 19:13:07.996686 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.954489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.11.18.65:46915)
0924 19:13:07.969060 (+ 14571us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.969082 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.971353 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.972050 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.972063 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.973082 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.974436 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.974449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.975932 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.975937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.978439 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.978455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.979794 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.979801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.980371 (+ 570us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.981484 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.981512 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.985458 (+ 3946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.989879 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.989891 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.989911 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.990609 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.991100 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.991103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.991107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.991369 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.992615 (+ 1246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.992618 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.993119 (+ 501us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.994585 (+ 1466us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.995224 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7467,"mutex_wait_us":637,"spinlock_wait_cycles":13056,"thread_start_us":14766,"threads_started":1}
I20260924 19:13:07.996771 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.961561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.11.18.65:46849)
0924 19:13:07.969709 (+ 8148us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.969725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.970999 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.972050 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.972063 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.973082 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.974435 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.974449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.975933 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.975937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.978449 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.978457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.979794 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.979801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.980364 (+ 563us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.981415 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.981446 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.985188 (+ 3742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.990007 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.990012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.990016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.990817 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.991250 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.991255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.991258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.991392 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.992546 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.992554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.993119 (+ 565us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.994989 (+ 1870us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.995404 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8028,"mutex_wait_us":1067,"spinlock_wait_cycles":49536,"thread_start_us":6108,"threads_started":1}
I20260924 19:13:07.998730 18476 heartbeater.cc:347] Connected to a master server at 127.11.18.124:37469
I20260924 19:13:07.999004 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.953175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:55551 (local address 127.11.18.126:34731)
0924 19:13:07.967067 (+ 13892us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:07.967074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:07.969117 (+ 2043us) server_negotiation.cc:407] Connection header received
0924 19:13:07.971579 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:07.971586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:07.971672 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:07.971768 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:07.974330 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.975445 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.978716 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.979375 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.979485 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.987145 (+ 7660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:07.987151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:07.987164 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:07.987212 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:07.990523 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.991446 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.991454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.991458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.991593 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:07.997210 (+ 5617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:07.997219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:07.997226 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:07.997624 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:07.997761 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:07.998397 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 19:13:07.998631 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13771,"thread_start_us":13673,"threads_started":1}
I20260924 19:13:07.999028 18476 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:07.999791 18476 heartbeater.cc:511] Master 127.11.18.124:37469 requested a full tablet report, sending...
I20260924 19:13:08.000003 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:07.961014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:55551)
0924 19:13:07.968879 (+ 7865us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:07.968967 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:07.970970 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:07.972051 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:07.972063 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:07.972993 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:07.974103 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.974129 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.975693 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.975698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.978466 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:07.978484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:07.979865 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:07.979871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:07.980358 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:07.981770 (+ 1412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:07.981809 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:07.986907 (+ 5098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:07.990686 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.990693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.990697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.991309 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.991736 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:07.991740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:07.991743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:07.997050 (+ 5307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:07.997843 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:07.997848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:07.998069 (+ 221us) client_negotiation.cc:769] Sending connection context
0924 19:13:07.999173 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0924 19:13:07.999511 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7402,"mutex_wait_us":352,"spinlock_wait_cycles":22016,"thread_start_us":917,"threads_started":1}
I20260924 19:13:08.000626 18478 heartbeater.cc:347] Connected to a master server at 127.11.18.125:35333
I20260924 19:13:08.000751 18478 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:08.000977 18477 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34731
I20260924 19:13:08.001119 18477 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:08.001202 18478 heartbeater.cc:511] Master 127.11.18.125:35333 requested a full tablet report, sending...
I20260924 19:13:08.001636 18477 heartbeater.cc:511] Master 127.11.18.126:34731 requested a full tablet report, sending...
I20260924 19:13:08.002841 17507 ts_manager.cc:194] Re-registered known tserver with Master: 8264c37ea70a460e94c0be82040316c4 (127.11.18.65:37241)
I20260924 19:13:08.003348 18298 ts_manager.cc:194] Registered new tserver with Master: 8264c37ea70a460e94c0be82040316c4 (127.11.18.65:37241)
I20260924 19:13:08.003465 18108 ts_manager.cc:194] Registered new tserver with Master: 8264c37ea70a460e94c0be82040316c4 (127.11.18.65:37241)
I20260924 19:13:08.004027 17507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:55551
I20260924 19:13:08.082104 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 17703
I20260924 19:13:08.100534 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:40157
--local_ip_for_outbound_sockets=127.11.18.66
--tserver_master_addrs=127.11.18.126:34731
--webserver_port=34165
--webserver_interface=127.11.18.66
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab
--principal=oryx/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333,127.11.18.124:37469 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:13:08.340013 18256 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: NON_VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: true } } }
I20260924 19:13:08.340473 18256 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:08.342128 18489 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: attempting to promote NON_VOTER a93be477dd93422ea19b9d8d06c25c69 to VOTER
I20260924 19:13:08.342703 18489 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } }
I20260924 19:13:08.344904 18308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 LEARNER]: Refusing update from remote peer d61771aac952491d8473090ebd0863c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 19:13:08.345053 18118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Refusing update from remote peer d61771aac952491d8473090ebd0863c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 19:13:08.345558 18490 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 19:13:08.346040 18489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 19:13:08.349247 18489 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:08.349946 18118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:08.353394 18256 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:08.354382 18256 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:08.355232 18490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:08.357455 18490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:08.356763 18308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:08.363876 18489 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:08.364243 18489 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:08.364918 18497 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:08.365728 18496 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d61771aac952491d8473090ebd0863c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:08.366179 18496 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:13:08.366359 18497 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:13:08.452491 18488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:08.452773 18488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:08.452867 18488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:08.460165 18488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:08.460290 18488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:08.460430 18488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:08.470466 18488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab
--principal=oryx/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:40157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=34165
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333,127.11.18.124:37469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:08.471802 18488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:08.473026 18488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:08.486784 18503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:08.486764 18502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:08.487164 18505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:08.488200 18488 server_base.cc:1034] running on GCE node
I20260924 19:13:08.489151 18488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:08.490345 18488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:08.491627 18488 hybrid_clock.cc:648] HybridClock initialized: now 1790277188491565 us; error 59 us; skew 500 ppm
I20260924 19:13:08.491990 18488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:08 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277188, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:08.495674 18488 init.cc:377] Logged in from keytab as oryx/127.11.18.66@KRBTEST.COM (short username oryx)
I20260924 19:13:08.497277 18488 webserver.cc:460] Webserver started at http://127.11.18.66:34165/ using document root <none> and password file <none>
I20260924 19:13:08.498059 18488 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:08.498224 18488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:08.504055 18488 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:13:08.507145 18512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:08.508948 18488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:13:08.509178 18488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "bb0d01f02d064ea0bbb746e824d86c52"
format_stamp: "Formatted at 2026-09-24 19:13:05 on dist-test-slave-0lgr"
server_key: "f84c2f22879177e773a3b5e266a25e9e"
server_key_iv: "dd2ed8f0b08ccca516ff9bede40b496a"
server_key_version: "encryptionkey@0"
I20260924 19:13:08.510522 18488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:08.528921 18488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:08.533802 18488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:08.534108 18488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:08.535221 18488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:08.536847 18488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:08.536978 18488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:08.537129 18488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:08.537226 18488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:08.574095 18488 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:40157
I20260924 19:13:08.574213 18624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:40157 every 8 connection(s)
I20260924 19:13:08.575901 18488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:08.583530 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 18488
I20260924 19:13:08.585448 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 17837
Sep 24 19:13:08 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277188, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.125@KRBTEST.COM
Sep 24 19:13:08 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.67@KRBTEST.COM
Sep 24 19:13:08 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277188, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
Sep 24 19:13:08 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277188, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.124@KRBTEST.COM
I20260924 19:13:08.654006 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:08.587227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.11.18.67:37589)
0924 19:13:08.620416 (+ 33189us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:08.620428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:08.620471 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:13:08.620568 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:08.620580 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:08.620973 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:08.621189 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:08.626442 (+ 5253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.628562 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.632641 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.633896 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.634061 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:08.644474 (+ 10413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:08.644498 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:08.644517 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:08.644560 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:08.649199 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:08.650350 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:08.650361 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:08.650365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:08.650568 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:08.651167 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:08.651172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:08.651175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:08.651610 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:08.651969 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:08.653197 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0924 19:13:08.653509 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32919,"thread_start_us":34117,"threads_started":1}
I20260924 19:13:08.654181 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:08.581408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:44283 (local address 127.11.18.125:35333)
0924 19:13:08.623368 (+ 41960us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:08.623379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:08.623414 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:13:08.623505 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:08.623514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:08.623666 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:08.623830 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:08.627951 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.629730 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.636223 (+ 6493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.637386 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.637502 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:08.642993 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:08.643000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:08.643005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:08.643050 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:08.647613 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:08.648842 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:08.648851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:08.648856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:08.649012 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:08.649645 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:08.649654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:08.649661 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:08.650137 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:08.650297 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:08.653539 (+ 3242us) server_negotiation.cc:299] Negotiation successful
0924 19:13:08.653801 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41773,"thread_start_us":46285,"threads_started":1}
I20260924 19:13:08.654590 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:08.580692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.11.18.66:44283)
0924 19:13:08.622533 (+ 41841us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:08.622569 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:08.622758 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:08.624662 (+ 1904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:08.624683 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:08.626148 (+ 1465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:08.627462 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:08.627500 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.630094 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.630104 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:08.633460 (+ 3356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:08.633484 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.637956 (+ 4472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.637965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:08.638354 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:08.639201 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:08.639233 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:08.642709 (+ 3476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:08.647816 (+ 5107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:08.647827 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:08.647855 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:08.648666 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:08.649158 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:08.649167 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:08.649172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:08.649486 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:08.650409 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:08.650419 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:08.650997 (+ 578us) client_negotiation.cc:769] Sending connection context
0924 19:13:08.651403 (+ 406us) client_negotiation.cc:240] Negotiation successful
0924 19:13:08.651997 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41645,"thread_start_us":41879,"threads_started":1}
I20260924 19:13:08.656045 17960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:08.656198 18627 heartbeater.cc:347] Connected to a master server at 127.11.18.125:35333
I20260924 19:13:08.656520 18627 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:08.657498 18627 heartbeater.cc:511] Master 127.11.18.125:35333 requested a full tablet report, sending...
I20260924 19:13:08.659863 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:08.579241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:34815 (local address 127.11.18.124:37469)
0924 19:13:08.629979 (+ 50738us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:08.629989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:08.630017 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:13:08.633846 (+ 3829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:08.633870 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:08.634002 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:08.634136 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:08.635994 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.637275 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.639984 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.640773 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.642333 (+ 1560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:08.648586 (+ 6253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:08.648595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:08.648600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:08.648646 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:08.656648 (+ 8002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:08.657583 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:08.657589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:08.657593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:08.657739 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:08.658233 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:08.658238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:08.658241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:08.658595 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:08.658725 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:08.659248 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 19:13:08.659516 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50545,"thread_start_us":50780,"threads_started":1}
I20260924 19:13:08.659925 18108 ts_manager.cc:194] Registered new tserver with Master: bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157)
I20260924 19:13:08.660822 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:08.579929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.11.18.66:34815)
0924 19:13:08.616478 (+ 36549us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:08.616626 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:08.621233 (+ 4607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:08.634422 (+ 13189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:08.634431 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:08.635252 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:08.635731 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:08.635750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.637540 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.637545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:08.639759 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:08.639779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.642657 (+ 2878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.642667 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:08.643085 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:08.644139 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:08.644171 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:08.648296 (+ 4125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:08.656820 (+ 8524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:08.656824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:08.656828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:08.657420 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:08.657889 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:08.657895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:08.657899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:08.658070 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:08.658819 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:08.658823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:08.658955 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:13:08.660048 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0924 19:13:08.660389 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35994,"mutex_wait_us":91,"thread_start_us":55313,"threads_started":1}
I20260924 19:13:08.661770 18625 heartbeater.cc:347] Connected to a master server at 127.11.18.124:37469
I20260924 19:13:08.661939 18625 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:08.662263 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:08.579930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:49573)
0924 19:13:08.635816 (+ 55886us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:08.635840 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:08.635961 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:08.637076 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:08.637082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:08.637821 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:08.638335 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:08.638347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.639931 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.639936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:08.641550 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:08.641563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.642722 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.642731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:08.643018 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:08.643845 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:08.643862 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:08.647616 (+ 3754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:08.652338 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:08.652344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:08.652350 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:08.652899 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:08.653523 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:08.653528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:08.653534 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:08.653685 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:08.655030 (+ 1345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:08.655035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:08.661238 (+ 6203us) client_negotiation.cc:769] Sending connection context
0924 19:13:08.661415 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 19:13:08.661734 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55688,"mutex_wait_us":3176,"thread_start_us":38308,"threads_started":1}
I20260924 19:13:08.662490 18625 heartbeater.cc:511] Master 127.11.18.124:37469 requested a full tablet report, sending...
I20260924 19:13:08.663269 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:08.579236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:49573 (local address 127.11.18.126:34731)
0924 19:13:08.636180 (+ 56944us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:08.636186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:08.636203 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:08.636266 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:08.636271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:08.636354 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:08.636450 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:08.638527 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.639754 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.641706 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:08.642320 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:08.642443 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:08.647886 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:08.647896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:08.647902 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:08.647972 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:08.652174 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:08.653200 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:08.653207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:08.653210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:08.653384 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:08.653810 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:08.653815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:08.653818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:08.654648 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:08.655059 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:08.662102 (+ 7043us) server_negotiation.cc:299] Negotiation successful
0924 19:13:08.662881 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56757,"thread_start_us":56623,"threads_started":1}
I20260924 19:13:08.663755 18298 ts_manager.cc:194] Registered new tserver with Master: bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157)
I20260924 19:13:08.664345 18626 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34731
I20260924 19:13:08.664526 18626 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:08.665148 18626 heartbeater.cc:511] Master 127.11.18.126:34731 requested a full tablet report, sending...
I20260924 19:13:08.666458 17507 ts_manager.cc:194] Re-registered known tserver with Master: bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157)
I20260924 19:13:08.667655 17507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:49573
I20260924 19:13:08.755537 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 17837
I20260924 19:13:08.771937 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:37589
--local_ip_for_outbound_sockets=127.11.18.67
--tserver_master_addrs=127.11.18.126:34731
--webserver_port=38975
--webserver_interface=127.11.18.67
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab
--principal=oryx/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333,127.11.18.124:37469 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:13:09.007332 18477 heartbeater.cc:503] Master 127.11.18.126:34731 was elected leader, sending a full tablet report...
W20260924 19:13:09.116670 18638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:09.117007 18638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:09.117177 18638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:09.124751 18638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:09.124919 18638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:09.125115 18638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:09.135334 18638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab
--principal=oryx/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:37589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=38975
--tserver_master_addrs=127.11.18.126:34731,127.11.18.125:35333,127.11.18.124:37469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:09.136770 18638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:09.138198 18638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:09.152227 18644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:09.152275 18643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:09.152352 18646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:09.153298 18638 server_base.cc:1034] running on GCE node
I20260924 19:13:09.153992 18638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:09.155048 18638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:09.156286 18638 hybrid_clock.cc:648] HybridClock initialized: now 1790277189156220 us; error 62 us; skew 500 ppm
I20260924 19:13:09.156615 18638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:09 dist-test-slave-0lgr krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277189, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:09.160135 18638 init.cc:377] Logged in from keytab as oryx/127.11.18.67@KRBTEST.COM (short username oryx)
I20260924 19:13:09.161832 18638 webserver.cc:460] Webserver started at http://127.11.18.67:38975/ using document root <none> and password file <none>
I20260924 19:13:09.162667 18638 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:09.162854 18638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:09.168220 18638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.002s
I20260924 19:13:09.171152 18653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:09.172639 18638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:13:09.172864 18638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "80c8923bc3f947d79e290801c4f0ee8c"
format_stamp: "Formatted at 2026-09-24 19:13:05 on dist-test-slave-0lgr"
server_key: "f4cd3bb1ef3f3b4800308ba6eae4d3a1"
server_key_iv: "daa3fe364911766cf82c0e0fe8725431"
server_key_version: "encryptionkey@0"
I20260924 19:13:09.174094 18638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:09.219677 18638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:09.224759 18638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:09.225096 18638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:09.226213 18638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:09.227782 18638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:09.227921 18638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:09.228075 18638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:09.228175 18638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:09.265789 18638 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:37589
I20260924 19:13:09.265892 18765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:37589 every 8 connection(s)
I20260924 19:13:09.267459 18638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:09.274899 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 18638
Sep 24 19:13:09 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277189, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.124@KRBTEST.COM
Sep 24 19:13:09 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277189, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.125@KRBTEST.COM
Sep 24 19:13:09 dist-test-slave-0lgr krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277189, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:09.306376 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.270594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:37505 (local address 127.11.18.124:37469)
0924 19:13:09.271153 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.271164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.277682 (+ 6518us) server_negotiation.cc:407] Connection header received
0924 19:13:09.278047 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.278056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.278201 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.278306 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:09.281438 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.283010 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.286375 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.286992 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.287084 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.296828 (+ 9744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:09.296837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:09.296844 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:09.296889 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:09.300806 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.302005 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.302012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.302016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.302157 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.302741 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.302746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.302750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.303995 (+ 1245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:09.304145 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.305212 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.305547 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"spinlock_wait_cycles":1280,"thread_start_us":356,"threads_started":1}
I20260924 19:13:09.309453 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.272452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.11.18.67:37505)
0924 19:13:09.277555 (+ 5103us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:09.277578 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:09.277805 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:09.278973 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:09.278984 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:09.280586 (+ 1602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:09.281189 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.281219 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.283275 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.283283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.286192 (+ 2909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.286210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.287498 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.287513 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.289780 (+ 2267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.290877 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:09.290909 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:09.295529 (+ 4620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:09.300995 (+ 5466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:09.301010 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:09.301038 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:09.301851 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:09.302303 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:09.302309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:09.302313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:09.302601 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:09.304246 (+ 1645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:09.304254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:09.304747 (+ 493us) client_negotiation.cc:769] Sending connection context
0924 19:13:09.307218 (+ 2471us) client_negotiation.cc:240] Negotiation successful
0924 19:13:09.307856 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4935,"mutex_wait_us":59,"thread_start_us":5092,"threads_started":1}
I20260924 19:13:09.311863 18766 heartbeater.cc:347] Connected to a master server at 127.11.18.124:37469
I20260924 19:13:09.312214 18766 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:09.313134 18766 heartbeater.cc:511] Master 127.11.18.124:37469 requested a full tablet report, sending...
I20260924 19:13:09.315317 18298 ts_manager.cc:194] Registered new tserver with Master: 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589)
I20260924 19:13:09.317373 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.271200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.11.18.67:55517)
0924 19:13:09.272306 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:09.272370 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:09.275719 (+ 3349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:09.278969 (+ 3250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:09.278982 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:09.280043 (+ 1061us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:09.281679 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.281697 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.288669 (+ 6972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.288675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.290387 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.290404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.293723 (+ 3319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.293729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.293901 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.294486 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:09.294508 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:09.297279 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:09.310180 (+ 12901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:09.310185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:09.310188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:09.310711 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:09.313922 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:09.313929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:09.313935 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:09.314113 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:09.316146 (+ 2033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:09.316152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:09.316329 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 19:13:09.316472 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 19:13:09.316820 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":40,"thread_start_us":5044,"threads_started":1}
I20260924 19:13:09.318277 18773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.271944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:55517 (local address 127.11.18.125:35333)
0924 19:13:09.278367 (+ 6423us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.278374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.278393 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:13:09.278461 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.278467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.278548 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.279857 (+ 1309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:09.286104 (+ 6247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.287206 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.290602 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.291306 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.294270 (+ 2964us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.297535 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:09.297543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:09.297548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:09.297595 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:09.305152 (+ 7557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.310915 (+ 5763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.310921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.310924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.311100 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.314303 (+ 3203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.314309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.314312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.314676 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:09.317660 (+ 2984us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.317789 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.317999 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6297,"thread_start_us":6166,"threads_started":1}
I20260924 19:13:09.319736 18768 heartbeater.cc:347] Connected to a master server at 127.11.18.125:35333
I20260924 19:13:09.319897 18768 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:09.320470 18768 heartbeater.cc:511] Master 127.11.18.125:35333 requested a full tablet report, sending...
I20260924 19:13:09.321724 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.272807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:34353)
0924 19:13:09.277816 (+ 5009us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:09.277831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:09.277928 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:09.279009 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:09.279013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:09.280031 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:09.287626 (+ 7595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.287646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.289503 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.289511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.293359 (+ 3848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.293373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.300357 (+ 6984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.300364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.300589 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.301765 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:09.301800 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:09.309173 (+ 7373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:09.314834 (+ 5661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:09.314839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:09.314842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:09.315472 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:09.319798 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:09.319802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:09.319827 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:09.320037 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:09.320640 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:09.320645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:09.320830 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 19:13:09.320972 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 19:13:09.321274 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4903,"mutex_wait_us":2130,"spinlock_wait_cycles":16768,"thread_start_us":4436,"threads_started":1}
I20260924 19:13:09.321717 18108 ts_manager.cc:194] Registered new tserver with Master: 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589)
I20260924 19:13:09.322137 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.270872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:34353 (local address 127.11.18.126:34731)
0924 19:13:09.276371 (+ 5499us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.276376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.277916 (+ 1540us) server_negotiation.cc:407] Connection header received
0924 19:13:09.278625 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.278630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.278714 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.278826 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:09.287863 (+ 9037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.289251 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.293508 (+ 4257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.294257 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.294395 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.309470 (+ 15075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:09.309480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:09.309487 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:09.309537 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:09.313965 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.315612 (+ 1647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.315621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.315628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.315861 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.320177 (+ 4316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.320183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.320186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.320506 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:09.320665 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.321360 (+ 695us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.321604 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5399,"thread_start_us":5319,"threads_started":1}
I20260924 19:13:09.323035 18767 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34731
I20260924 19:13:09.323160 18767 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:09.323596 18767 heartbeater.cc:511] Master 127.11.18.126:34731 requested a full tablet report, sending...
I20260924 19:13:09.324586 17507 ts_manager.cc:194] Re-registered known tserver with Master: 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589)
I20260924 19:13:09.325913 17507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:34353
I20260924 19:13:09.339128 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.281236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.11.18.126:34731)
0924 19:13:09.307605 (+ 26369us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.307611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.308211 (+ 600us) server_negotiation.cc:407] Connection header received
0924 19:13:09.308402 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.308412 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.308522 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.308660 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:09.314188 (+ 5528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.316169 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.323848 (+ 7679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.324608 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.324837 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.331100 (+ 6263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:09.331108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:09.331113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:09.331160 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:09.335011 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.336593 (+ 1582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.336598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.336601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.336761 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.337315 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.337373 (+ 58us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.337380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.337802 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:09.338046 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.338456 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.338710 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26223,"thread_start_us":26125,"threads_started":1}
I20260924 19:13:09.339964 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.282045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.11.18.125:35333)
0924 19:13:09.306330 (+ 24285us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.306338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.316752 (+ 10414us) server_negotiation.cc:407] Connection header received
0924 19:13:09.316803 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.316807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.316867 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.316946 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:09.322423 (+ 5477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.323630 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.325769 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.326449 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.326583 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.332203 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:09.332210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:09.332214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:09.332257 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:09.335741 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.336553 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.336561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.336567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.336761 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.337540 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.337545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.337549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.337943 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:09.338151 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.338893 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.339121 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24132,"thread_start_us":23987,"threads_started":1}
I20260924 19:13:09.340528 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.296441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.11.18.124:37469)
0924 19:13:09.307139 (+ 10698us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.307147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.319493 (+ 12346us) server_negotiation.cc:407] Connection header received
0924 19:13:09.319737 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.319743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.319836 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.319950 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:09.326301 (+ 6351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.327636 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.329541 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.330220 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.330378 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.334292 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:09.334297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:09.334301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:09.334342 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:09.337711 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.338520 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.338526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.338529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.338683 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.339081 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.339086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.339089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.339380 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:09.339625 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.339980 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.340186 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10539,"thread_start_us":20883,"threads_started":1}
I20260924 19:13:09.344223 17507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:13:09.351598 17507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:13:09.387048 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.371111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:37241 (local address 127.0.0.1:49704)
0924 19:13:09.372644 (+ 1533us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:09.372669 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:09.372784 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:09.378763 (+ 5979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:09.378772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:09.378809 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:09.379351 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.379366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.382131 (+ 2765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.382137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.384479 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.384491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.386053 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.386060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.386294 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.386338 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:13:09.386428 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:13:09.386583 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"thread_start_us":11241,"threads_started":1}
I20260924 19:13:09.387542 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.371747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.11.18.65:37241)
0924 19:13:09.376129 (+ 4382us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.376142 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.376183 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:13:09.376349 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.376361 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.376745 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.379642 (+ 2897us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:09.379941 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.381959 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.384643 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.385857 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.386027 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.386672 (+ 645us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.386763 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.386941 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4168,"thread_start_us":4012,"threads_started":1}
I20260924 19:13:09.387953 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.370837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:37589 (local address 127.0.0.1:45848)
0924 19:13:09.371883 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:09.371909 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:09.372058 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:09.378475 (+ 6417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:09.378481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:09.378513 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:09.379049 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.379063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.380856 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.380864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.383417 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.383435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.386337 (+ 2902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.386343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.386542 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.386656 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 19:13:09.386772 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:13:09.386951 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":52,"thread_start_us":702,"threads_started":1}
I20260924 19:13:09.390153 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.371145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.11.18.67:37589)
0924 19:13:09.377588 (+ 6443us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.377600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.377640 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:13:09.377753 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.377764 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.378045 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.378327 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:09.379273 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.380698 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.384350 (+ 3652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.386108 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.386269 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.386477 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.389619 (+ 3142us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.389818 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6213,"thread_start_us":5984,"threads_started":1}
I20260924 19:13:09.392035 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.371938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:40157 (local address 127.0.0.1:32896)
0924 19:13:09.382665 (+ 10727us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:09.382682 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:09.382848 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:09.383706 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:09.383711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:09.383734 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:09.384123 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.384156 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.386555 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.386558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.389362 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.389377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.391238 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.391244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.391427 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.391464 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:13:09.391566 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:13:09.391700 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10576,"thread_start_us":369,"threads_started":1}
I20260924 19:13:09.392088 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.372432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.11.18.66:40157)
0924 19:13:09.373353 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.373367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.382806 (+ 9439us) server_negotiation.cc:407] Connection header received
0924 19:13:09.383025 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.383033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.383284 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.383484 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:09.384288 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.385775 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.389535 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.391010 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.391153 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.391366 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.391624 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.391801 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":569,"threads_started":1}
I20260924 19:13:09.392068 18411 tablet_service.cc:1467] Processing CreateTablet for tablet ebd45b31bb31450bb39641e621dd26d2 (DEFAULT_TABLE table=test-table [id=1d4c94ddce354ab889be5c0ccc36bd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:09.393558 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd45b31bb31450bb39641e621dd26d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:09.394168 18701 tablet_service.cc:1467] Processing CreateTablet for tablet ebd45b31bb31450bb39641e621dd26d2 (DEFAULT_TABLE table=test-table [id=1d4c94ddce354ab889be5c0ccc36bd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:09.395591 18701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd45b31bb31450bb39641e621dd26d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:09.395851 18560 tablet_service.cc:1467] Processing CreateTablet for tablet ebd45b31bb31450bb39641e621dd26d2 (DEFAULT_TABLE table=test-table [id=1d4c94ddce354ab889be5c0ccc36bd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:09.397274 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd45b31bb31450bb39641e621dd26d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:09.406440 18801 tablet_bootstrap.cc:492] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Bootstrap starting.
I20260924 19:13:09.410245 18801 tablet_bootstrap.cc:654] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:09.411716 18801 log.cc:824] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:09.412943 18803 tablet_bootstrap.cc:492] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Bootstrap starting.
I20260924 19:13:09.415329 18802 tablet_bootstrap.cc:492] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Bootstrap starting.
I20260924 19:13:09.417101 18803 tablet_bootstrap.cc:654] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:09.417588 18801 tablet_bootstrap.cc:492] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: No bootstrap required, opened a new log
I20260924 19:13:09.417862 18801 ts_tablet_manager.cc:1397] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260924 19:13:09.418555 18803 log.cc:824] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:09.421169 18802 tablet_bootstrap.cc:654] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:09.422017 18803 tablet_bootstrap.cc:492] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: No bootstrap required, opened a new log
I20260924 19:13:09.422281 18803 ts_tablet_manager.cc:1397] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 19:13:09.422588 18802 log.cc:824] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:09.425465 18802 tablet_bootstrap.cc:492] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: No bootstrap required, opened a new log
I20260924 19:13:09.425757 18802 ts_tablet_manager.cc:1397] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260924 19:13:09.429108 18801 raft_consensus.cc:348] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.429618 18801 raft_consensus.cc:374] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:09.429723 18801 raft_consensus.cc:729] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8264c37ea70a460e94c0be82040316c4, State: Initialized, Role: FOLLOWER
I20260924 19:13:09.430399 18801 consensus_queue.cc:260] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.432238 18801 ts_tablet_manager.cc:1428] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 19:13:09.433182 18803 raft_consensus.cc:348] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.433645 18803 raft_consensus.cc:374] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:09.433745 18803 raft_consensus.cc:729] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0d01f02d064ea0bbb746e824d86c52, State: Initialized, Role: FOLLOWER
I20260924 19:13:09.434455 18803 consensus_queue.cc:260] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.437829 18802 raft_consensus.cc:348] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.438511 18802 raft_consensus.cc:374] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:09.438697 18802 raft_consensus.cc:729] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c8923bc3f947d79e290801c4f0ee8c, State: Initialized, Role: FOLLOWER
I20260924 19:13:09.439524 18802 consensus_queue.cc:260] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.441648 18626 heartbeater.cc:503] Master 127.11.18.126:34731 was elected leader, sending a full tablet report...
I20260924 19:13:09.443141 18802 ts_tablet_manager.cc:1428] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260924 19:13:09.443660 18767 heartbeater.cc:503] Master 127.11.18.126:34731 was elected leader, sending a full tablet report...
I20260924 19:13:09.441417 18803 ts_tablet_manager.cc:1428] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
W20260924 19:13:09.469244 18479 tablet.cc:2369] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:13:09.525821 18769 tablet.cc:2369] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:13:09.579864 18628 tablet.cc:2369] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:13:09.658255 18807 raft_consensus.cc:482] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:09.658631 18807 raft_consensus.cc:504] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.660665 18807 leader_election.cc:288] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589), bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157)
I20260924 19:13:09.670744 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.661081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:46527 (local address 127.11.18.67:37589)
0924 19:13:09.661501 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.661512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.661983 (+ 471us) server_negotiation.cc:407] Connection header received
0924 19:13:09.662238 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.662244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.662313 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.662373 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:09.663207 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.664590 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.668080 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.669529 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.669743 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.669826 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.670342 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.670488 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:13:09.670918 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.661026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:37589 (local address 127.11.18.65:46527)
0924 19:13:09.661870 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:09.661889 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:09.662091 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:09.662583 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:09.662587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:09.662679 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:09.663046 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.663056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.664816 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.664823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.667843 (+ 3020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.667861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.669836 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.669844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.670066 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.670152 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 19:13:09.670260 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:13:09.670405 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":279,"thread_start_us":484,"threads_started":1}
I20260924 19:13:09.671150 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.662162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:34087 (local address 127.11.18.66:40157)
0924 19:13:09.662376 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.662385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.662801 (+ 416us) server_negotiation.cc:407] Connection header received
0924 19:13:09.663031 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.663036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.663097 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.663209 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:09.663899 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.664929 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.668080 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.669615 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.669764 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.669833 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.670624 (+ 791us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.670758 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 19:13:09.671233 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.662022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:40157 (local address 127.11.18.65:34087)
0924 19:13:09.662677 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:09.662691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:09.662873 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:09.663374 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:09.663378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:09.663398 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:09.663722 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.663734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.665075 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.665080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.667831 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:09.667850 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.669836 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.669844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:09.670065 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.670154 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 19:13:09.670341 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 19:13:09.670442 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":180,"thread_start_us":376,"threads_started":1}
I20260924 19:13:09.671928 18721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd45b31bb31450bb39641e621dd26d2" candidate_uuid: "8264c37ea70a460e94c0be82040316c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c8923bc3f947d79e290801c4f0ee8c" is_pre_election: true
I20260924 19:13:09.672156 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd45b31bb31450bb39641e621dd26d2" candidate_uuid: "8264c37ea70a460e94c0be82040316c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d01f02d064ea0bbb746e824d86c52" is_pre_election: true
I20260924 19:13:09.672458 18721 raft_consensus.cc:2454] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8264c37ea70a460e94c0be82040316c4 in term 0.
I20260924 19:13:09.672672 18580 raft_consensus.cc:2454] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8264c37ea70a460e94c0be82040316c4 in term 0.
I20260924 19:13:09.673125 18367 leader_election.cc:302] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c8923bc3f947d79e290801c4f0ee8c, 8264c37ea70a460e94c0be82040316c4; no voters:
I20260924 19:13:09.673573 18807 raft_consensus.cc:2781] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:09.673698 18807 raft_consensus.cc:482] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:09.673849 18807 raft_consensus.cc:3037] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:09.676328 18807 raft_consensus.cc:504] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.676901 18807 leader_election.cc:288] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [CANDIDATE]: Term 1 election: Requested vote from peers 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589), bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157)
I20260924 19:13:09.677594 18721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd45b31bb31450bb39641e621dd26d2" candidate_uuid: "8264c37ea70a460e94c0be82040316c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c8923bc3f947d79e290801c4f0ee8c"
I20260924 19:13:09.677594 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd45b31bb31450bb39641e621dd26d2" candidate_uuid: "8264c37ea70a460e94c0be82040316c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d01f02d064ea0bbb746e824d86c52"
I20260924 19:13:09.677811 18721 raft_consensus.cc:3037] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:09.677879 18580 raft_consensus.cc:3037] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:09.681505 18580 raft_consensus.cc:2454] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8264c37ea70a460e94c0be82040316c4 in term 1.
I20260924 19:13:09.681519 18721 raft_consensus.cc:2454] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8264c37ea70a460e94c0be82040316c4 in term 1.
I20260924 19:13:09.681968 18367 leader_election.cc:302] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8264c37ea70a460e94c0be82040316c4, bb0d01f02d064ea0bbb746e824d86c52; no voters:
I20260924 19:13:09.682274 18807 raft_consensus.cc:2781] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:09.682744 18807 raft_consensus.cc:686] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 1 LEADER]: Becoming Leader. State: Replica: 8264c37ea70a460e94c0be82040316c4, State: Running, Role: LEADER
I20260924 19:13:09.683497 18807 consensus_queue.cc:237] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:09.689648 17507 catalog_manager.cc:5495] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8264c37ea70a460e94c0be82040316c4 (127.11.18.65). New cstate: current_term: 1 leader_uuid: "8264c37ea70a460e94c0be82040316c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } health_report { overall_health: UNKNOWN } } }
I20260924 19:13:09.752264 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.746164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.11.18.65:37241)
0924 19:13:09.746400 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.746406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.746425 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:13:09.746657 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.746661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.746768 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.746983 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:09.747368 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.748548 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.750011 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.750495 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.750629 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.751094 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:09.751663 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:09.751811 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.751921 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.752000 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:13:09.760716 18580 raft_consensus.cc:1261] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 1 FOLLOWER]: Refusing update from remote peer 8264c37ea70a460e94c0be82040316c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:09.761015 18721 raft_consensus.cc:1261] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 1 FOLLOWER]: Refusing update from remote peer 8264c37ea70a460e94c0be82040316c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:09.761809 18807 consensus_queue.cc:1035] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:09.762117 18812 consensus_queue.cc:1035] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:09.775259 18816 mvcc.cc:205] Tried to move back new op lower bound from 7332975369248751616 to 7332975368947802112. Current Snapshot: MvccSnapshot[applied={T|T < 7332975369248751616}]
I20260924 19:13:09.781140 18818 mvcc.cc:205] Tried to move back new op lower bound from 7332975369248751616 to 7332975368947802112. Current Snapshot: MvccSnapshot[applied={T|T < 7332975369248751616}]
I20260924 19:13:09.792791 17508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:09.793129 17508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:09.801971 17508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d61771aac952491d8473090ebd0863c1: Sending DeleteTablet for 3 replicas of tablet ebd45b31bb31450bb39641e621dd26d2
I20260924 19:13:09.802986 18411 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd45b31bb31450bb39641e621dd26d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:09 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:49704
I20260924 19:13:09.804734 18701 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd45b31bb31450bb39641e621dd26d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:09 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:45848
I20260924 19:13:09.804818 18560 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd45b31bb31450bb39641e621dd26d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:09 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:32896
I20260924 19:13:09.805429 18559 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd45b31bb31450bb39641e621dd26d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:09 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:32896
I20260924 19:13:09.814445 18827 tablet_replica.cc:330] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: stopping tablet replica
I20260924 19:13:09.815172 18827 raft_consensus.cc:2229] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:09.815719 18827 raft_consensus.cc:2258] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:09.816511 18825 tablet_replica.cc:330] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: stopping tablet replica
I20260924 19:13:09.816906 18825 raft_consensus.cc:2229] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:13:09.818195 18825 raft_consensus.cc:2258] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:09.818362 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 17476
I20260924 19:13:09.819342 18828 tablet_replica.cc:330] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: stopping tablet replica
I20260924 19:13:09.819410 17538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:09.819765 18828 raft_consensus.cc:2229] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:09.820086 18827 ts_tablet_manager.cc:1905] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:09.820538 18828 raft_consensus.cc:2258] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:09.824685 18828 ts_tablet_manager.cc:1905] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:09.825084 18825 ts_tablet_manager.cc:1905] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:09.828174 18827 ts_tablet_manager.cc:1918] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:09.828356 18827 log.cc:1196] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/ebd45b31bb31450bb39641e621dd26d2
I20260924 19:13:09.828918 18827 ts_tablet_manager.cc:1939] T ebd45b31bb31450bb39641e621dd26d2 P 80c8923bc3f947d79e290801c4f0ee8c: Deleting consensus metadata
I20260924 19:13:09.831591 18825 ts_tablet_manager.cc:1918] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:09.831780 18825 log.cc:1196] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/ebd45b31bb31450bb39641e621dd26d2
I20260924 19:13:09.832273 18825 ts_tablet_manager.cc:1939] T ebd45b31bb31450bb39641e621dd26d2 P 8264c37ea70a460e94c0be82040316c4: Deleting consensus metadata
I20260924 19:13:09.832653 18828 ts_tablet_manager.cc:1918] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:09.832809 18828 log.cc:1196] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/ebd45b31bb31450bb39641e621dd26d2
I20260924 19:13:09.833354 18828 ts_tablet_manager.cc:1939] T ebd45b31bb31450bb39641e621dd26d2 P bb0d01f02d064ea0bbb746e824d86c52: Deleting consensus metadata
I20260924 19:13:09.847736 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.805608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.11.18.125:35333)
0924 19:13:09.805778 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.805784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.816937 (+ 11153us) server_negotiation.cc:407] Connection header received
0924 19:13:09.820559 (+ 3622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.820567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.820655 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.821148 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:09.826336 (+ 5188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.827400 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.835531 (+ 8131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.836332 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.836460 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.840476 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:09.840483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:09.840488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:09.840541 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:09.844156 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.845052 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.845059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.845063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.845228 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.845744 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.845750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.845756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.846115 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:09.846405 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.847028 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.847307 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:13:09.866532 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.817133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.11.18.124:37469)
0924 19:13:09.817308 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.817314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.849196 (+ 31882us) server_negotiation.cc:407] Connection header received
0924 19:13:09.849615 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.849622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.849713 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.849786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:09.851843 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.852964 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.855295 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:09.855954 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:09.856057 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:09.859893 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:09.859900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:09.859905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:09.859950 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:09.863350 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.864165 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.864172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.864175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.864331 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:09.864770 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:09.864776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:09.864781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:09.865131 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:09.865421 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:09.865903 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 19:13:09.866115 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:13:09.956559 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 17476
W20260924 19:13:09.985203 18516 connection.cc:535] server connection from 127.0.0.1:32896 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:13:09.987051 18655 connection.cc:535] server connection from 127.0.0.1:45848 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:13:10.016255 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.992564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.11.18.124:37469)
0924 19:13:09.992800 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.992809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.998346 (+ 5537us) server_negotiation.cc:407] Connection header received
0924 19:13:09.998583 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.998592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.998681 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.998748 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:10.000876 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:10.001917 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:10.003965 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:10.004431 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:10.004575 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:10.008703 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:10.008711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:10.008716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:10.008763 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:10.012534 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:10.013574 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:10.013580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:10.013584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:10.013730 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:10.014270 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:10.014276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:10.014280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:10.014730 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:10.015037 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:10.015608 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 19:13:10.015900 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:13:10.017311 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:09.990939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.11.18.125:35333)
0924 19:13:09.991113 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:09.991120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:09.997365 (+ 6245us) server_negotiation.cc:407] Connection header received
0924 19:13:09.998661 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:09.998668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:09.998774 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:09.998894 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:10.001396 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:10.002490 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:10.004018 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:10.004723 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:10.004876 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:10.009608 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:10.009617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:10.009624 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:10.009676 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:10.013639 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:10.014536 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:10.014542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:10.014546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:10.014730 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:10.015174 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:10.015181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:10.015185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:10.015483 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:10.015796 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:10.016221 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 19:13:10.016923 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:13:10.785233 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:10.784401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:58539)
0924 19:13:10.784889 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:10.785039 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":251,"threads_started":1}
W20260924 19:13:10.785919 18477 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:34731 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:10.787985 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:10.787017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:56167)
0924 19:13:10.787585 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:10.787764 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":323,"threads_started":1}
W20260924 19:13:10.788335 18767 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:34731 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:10.806704 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:10.806061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:51537)
0924 19:13:10.806444 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:10.806550 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":185,"threads_started":1}
W20260924 19:13:10.807132 18626 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:34731 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:11.302788 18840 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d61771aac952491d8473090ebd0863c1)
I20260924 19:13:11.302841 18841 raft_consensus.cc:482] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d61771aac952491d8473090ebd0863c1)
I20260924 19:13:11.303061 18841 raft_consensus.cc:504] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } }
I20260924 19:13:11.303086 18840 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } }
I20260924 19:13:11.304397 18840 leader_election.cc:288] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731), a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124:37469)
I20260924 19:13:11.304417 18841 leader_election.cc:288] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731), ddd1d7a338bb4df4b3777224dafcfd12 (127.11.18.125:35333)
I20260924 19:13:11.307276 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.304829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48428)
0924 19:13:11.306635 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.306942 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1629,"threads_started":1}
W20260924 19:13:11.319705 18282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:11.320360 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.304516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48422)
0924 19:13:11.319784 (+ 15268us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.319978 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15141,"thread_start_us":15519,"threads_started":1}
W20260924 19:13:11.321087 18092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:11.328939 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.328514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48446)
0924 19:13:11.328661 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.328750 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 19:13:11.329079 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.328368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48434)
0924 19:13:11.328569 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.328788 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 19:13:11.329272 18092 leader_election.cc:334] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731): Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
W20260924 19:13:11.329576 18282 leader_election.cc:334] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731): Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:11.329870 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.305475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:35333 (local address 127.0.0.1:52274)
0924 19:13:11.308755 (+ 3280us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.308781 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:11.308968 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:11.321554 (+ 12586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:11.321562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:11.321599 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:11.321995 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:11.322011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:11.324381 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:11.324389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:11.327211 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:11.327228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:11.329117 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:11.329123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:11.329312 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:11.329364 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 19:13:11.329442 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:13:11.329584 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3141,"thread_start_us":844,"threads_started":1}
I20260924 19:13:11.330338 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.305029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.11.18.124:37469)
0924 19:13:11.318890 (+ 13861us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:11.318900 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:11.320025 (+ 1125us) server_negotiation.cc:407] Connection header received
0924 19:13:11.320622 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:11.320630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:11.320731 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:11.320819 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:11.322035 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:11.323745 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:11.327315 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:11.328950 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:11.329064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:11.329183 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:11.329888 (+ 705us) server_negotiation.cc:299] Negotiation successful
0924 19:13:11.330016 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13648,"thread_start_us":13874,"threads_started":1}
I20260924 19:13:11.330351 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.319834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.11.18.125:35333)
0924 19:13:11.321109 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:11.321117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:11.321136 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:13:11.321218 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:11.321224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:11.321348 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:11.322207 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:11.322510 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:11.324165 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:11.327445 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:11.328913 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:11.329710 (+ 797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:11.329818 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:11.329864 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 19:13:11.329989 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":1141,"spinlock_wait_cycles":3584,"thread_start_us":978,"threads_started":1}
I20260924 19:13:11.330973 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.304610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:37469 (local address 127.0.0.1:60574)
0924 19:13:11.319881 (+ 15271us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.319915 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:11.320405 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:11.321209 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:11.321217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:11.321273 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:11.321750 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:11.321767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:11.323993 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:11.324002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:11.327106 (+ 3104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:11.327130 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:11.329385 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:11.329392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:11.329603 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:11.329711 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 19:13:11.330499 (+ 788us) client_negotiation.cc:240] Negotiation successful
0924 19:13:11.330634 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15158,"mutex_wait_us":39,"spinlock_wait_cycles":3840,"thread_start_us":14909,"threads_started":1}
I20260924 19:13:11.330957 18118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a93be477dd93422ea19b9d8d06c25c69" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" is_pre_election: true
I20260924 19:13:11.331334 18118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a93be477dd93422ea19b9d8d06c25c69 in term 1.
I20260924 19:13:11.331557 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a93be477dd93422ea19b9d8d06c25c69" is_pre_election: true
I20260924 19:13:11.331934 18308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddd1d7a338bb4df4b3777224dafcfd12 in term 1.
I20260924 19:13:11.331955 18284 leader_election.cc:302] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a93be477dd93422ea19b9d8d06c25c69, ddd1d7a338bb4df4b3777224dafcfd12; no voters: d61771aac952491d8473090ebd0863c1
I20260924 19:13:11.332440 18841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:13:11.332422 18093 leader_election.cc:302] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a93be477dd93422ea19b9d8d06c25c69, ddd1d7a338bb4df4b3777224dafcfd12; no voters: d61771aac952491d8473090ebd0863c1
I20260924 19:13:11.332571 18841 raft_consensus.cc:482] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d61771aac952491d8473090ebd0863c1)
I20260924 19:13:11.332683 18841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:13:11.332768 18840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:13:11.332890 18840 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d61771aac952491d8473090ebd0863c1)
I20260924 19:13:11.332976 18840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:13:11.334883 18841 raft_consensus.cc:504] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } }
I20260924 19:13:11.335063 18840 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } }
I20260924 19:13:11.335434 18841 leader_election.cc:288] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [CANDIDATE]: Term 2 election: Requested vote from peers d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731), ddd1d7a338bb4df4b3777224dafcfd12 (127.11.18.125:35333)
I20260924 19:13:11.335744 18840 leader_election.cc:288] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 2 election: Requested vote from peers d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731), a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124:37469)
I20260924 19:13:11.335963 18118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a93be477dd93422ea19b9d8d06c25c69" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ddd1d7a338bb4df4b3777224dafcfd12"
I20260924 19:13:11.336126 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.335624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48452)
0924 19:13:11.335875 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.335967 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260924 19:13:11.336234 18118 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a93be477dd93422ea19b9d8d06c25c69 in current term 2: Already voted for candidate ddd1d7a338bb4df4b3777224dafcfd12 in this term.
I20260924 19:13:11.336200 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a93be477dd93422ea19b9d8d06c25c69"
I20260924 19:13:11.336483 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.335831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48456)
0924 19:13:11.336238 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.336328 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
I20260924 19:13:11.336476 18308 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ddd1d7a338bb4df4b3777224dafcfd12 in current term 2: Already voted for candidate a93be477dd93422ea19b9d8d06c25c69 in this term.
I20260924 19:13:11.337203 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.336911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48466)
0924 19:13:11.337045 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.337112 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 19:13:11.337491 18282 leader_election.cc:334] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731): Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:11.337543 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.337198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48474)
0924 19:13:11.337364 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.337439 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:13:11.337646 18282 leader_election.cc:302] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a93be477dd93422ea19b9d8d06c25c69; no voters: d61771aac952491d8473090ebd0863c1, ddd1d7a338bb4df4b3777224dafcfd12
W20260924 19:13:11.337759 18092 leader_election.cc:334] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731): Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:11.337935 18841 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 19:13:11.337924 18092 leader_election.cc:302] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddd1d7a338bb4df4b3777224dafcfd12; no voters: a93be477dd93422ea19b9d8d06c25c69, d61771aac952491d8473090ebd0863c1
I20260924 19:13:11.338187 18840 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 19:13:11.787786 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.786762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:49703)
0924 19:13:11.787456 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.787562 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"threads_started":1}
I20260924 19:13:11.790014 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.789107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:47639)
0924 19:13:11.789774 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.789854 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":429,"threads_started":1}
I20260924 19:13:11.808571 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:11.807827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:37455)
0924 19:13:11.808271 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:11.808404 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
I20260924 19:13:12.791100 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:12.789132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:47009)
0924 19:13:12.790726 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:12.790835 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1452,"thread_start_us":1321,"threads_started":1}
I20260924 19:13:12.791639 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:12.790809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:47953)
0924 19:13:12.791272 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:12.791411 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260924 19:13:12.810576 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:12.809735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:47695)
0924 19:13:12.810279 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:12.810399 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":329,"threads_started":1}
I20260924 19:13:13.097918 18856 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:13.098155 18856 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } }
I20260924 19:13:13.098865 18856 leader_election.cc:288] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731), a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124:37469)
I20260924 19:13:13.099615 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a93be477dd93422ea19b9d8d06c25c69" is_pre_election: true
I20260924 19:13:13.099928 18308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddd1d7a338bb4df4b3777224dafcfd12 in term 2.
I20260924 19:13:13.099944 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.099012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48726)
0924 19:13:13.099607 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.099716 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":343,"threads_started":1}
I20260924 19:13:13.100457 18093 leader_election.cc:302] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a93be477dd93422ea19b9d8d06c25c69, ddd1d7a338bb4df4b3777224dafcfd12; no voters:
I20260924 19:13:13.100771 18856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 19:13:13.100924 18856 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:13.101037 18856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:13:13.101711 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.101120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48738)
0924 19:13:13.101272 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.101544 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:13:13.102023 18092 leader_election.cc:334] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731): Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:13.103546 18856 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } }
I20260924 19:13:13.104120 18856 leader_election.cc:288] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 3 election: Requested vote from peers d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731), a93be477dd93422ea19b9d8d06c25c69 (127.11.18.124:37469)
I20260924 19:13:13.104548 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.104200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48752)
0924 19:13:13.104350 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.104427 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 19:13:13.104746 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a93be477dd93422ea19b9d8d06c25c69"
I20260924 19:13:13.104979 18308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:13:13.105619 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.105232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48756)
0924 19:13:13.105410 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.105487 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:13:13.105830 18092 leader_election.cc:334] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731): Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
I20260924 19:13:13.107292 18308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddd1d7a338bb4df4b3777224dafcfd12 in term 3.
I20260924 19:13:13.107749 18093 leader_election.cc:302] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a93be477dd93422ea19b9d8d06c25c69, ddd1d7a338bb4df4b3777224dafcfd12; no voters: d61771aac952491d8473090ebd0863c1
I20260924 19:13:13.108062 18856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:13:13.108570 18856 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [term 3 LEADER]: Becoming Leader. State: Replica: ddd1d7a338bb4df4b3777224dafcfd12, State: Running, Role: LEADER
I20260924 19:13:13.109051 18856 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } }
I20260924 19:13:13.111127 18859 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddd1d7a338bb4df4b3777224dafcfd12. Latest consensus state: current_term: 3 leader_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:13.111528 18859 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:13.112239 18861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:13:13.115928 18861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:13:13.116736 18861 catalog_manager.cc:1259] Loaded cluster ID: c345268d6d3a479aa0c4616a08853079
I20260924 19:13:13.116813 18861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:13:13.118150 18861 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:13:13.119164 18861 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Loaded TSK: 0
I20260924 19:13:13.119969 18861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:13:13.138238 18108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:13:13.145254 18108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:13:13.150429 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.149875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48772)
0924 19:13:13.150109 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.150278 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 19:13:13.150395 18308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [term 3 FOLLOWER]: Refusing update from remote peer ddd1d7a338bb4df4b3777224dafcfd12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 19:13:13.151213 18859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 19:13:13.151538 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.151127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48788)
0924 19:13:13.151301 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.151377 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:13:13.151731 18092 consensus_peers.cc:487] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 -> Peer d61771aac952491d8473090ebd0863c1 (127.11.18.126:34731): Couldn't send request to peer d61771aac952491d8473090ebd0863c1. Status: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:13:13.154929 18863 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddd1d7a338bb4df4b3777224dafcfd12. Latest consensus state: current_term: 3 leader_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:13.155392 18863 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:13:13.156201 18856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:13.156594 18856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:13.158964 18864 mvcc.cc:205] Tried to move back new op lower bound from 7332975383134490624 to 7332975382978465792. Current Snapshot: MvccSnapshot[applied={T|T < 7332975383134490624}]
I20260924 19:13:13.165150 18863 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d61771aac952491d8473090ebd0863c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34731 } } peers { permanent_uuid: "ddd1d7a338bb4df4b3777224dafcfd12" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 35333 } attrs { promote: false } } peers { permanent_uuid: "a93be477dd93422ea19b9d8d06c25c69" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 37469 } attrs { promote: false } } }
I20260924 19:13:13.165469 18863 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93be477dd93422ea19b9d8d06c25c69 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:13:13.183434 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.170947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.11.18.65:37241)
0924 19:13:13.176089 (+ 5142us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:13.176095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:13.176116 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:13:13.176276 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:13.176282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:13.176350 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:13.176434 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:13.177231 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.178371 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.181074 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.182333 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.182487 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.182601 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:13.182902 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 19:13:13.183051 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5002,"thread_start_us":4664,"threads_started":1}
I20260924 19:13:13.183522 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.170299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:37241 (local address 127.0.0.1:49722)
0924 19:13:13.170487 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.170515 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:13.170672 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:13.176695 (+ 6023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:13.176701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:13.176724 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:13.177026 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:13.177039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.178523 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.178528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:13.180942 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:13.180954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.182504 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.182510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:13.182727 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.182754 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 19:13:13.182876 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 19:13:13.183025 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":6}
I20260924 19:13:13.184079 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.172503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.11.18.66:40157)
0924 19:13:13.173156 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:13.173165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:13.173923 (+ 758us) server_negotiation.cc:407] Connection header received
0924 19:13:13.174258 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:13.174268 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:13.174384 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:13.174479 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:13.175510 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.177984 (+ 2474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.180965 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.182352 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.182488 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.182601 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:13.183600 (+ 999us) server_negotiation.cc:299] Negotiation successful
0924 19:13:13.183750 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":309,"threads_started":1}
I20260924 19:13:13.184514 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.172158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:40157 (local address 127.0.0.1:32910)
0924 19:13:13.173802 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.173827 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:13.174048 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:13.174789 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:13.174798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:13.174834 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:13.175264 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:13.175279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.178155 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.178164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:13.180707 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:13.180722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.182745 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.182752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:13.182965 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.183001 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:13:13.183090 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:13:13.183372 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"mutex_wait_us":72,"thread_start_us":1374,"threads_started":1}
I20260924 19:13:13.184624 18411 tablet_service.cc:1467] Processing CreateTablet for tablet 16585373e3044b42b5d546c3d4d3f6f1 (DEFAULT_TABLE table=test-table [id=9a354661f0b244b195feb6e906e76111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:13.185191 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16585373e3044b42b5d546c3d4d3f6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:13.185428 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.170634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:37589 (local address 127.0.0.1:45852)
0924 19:13:13.172086 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.172105 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:13.172584 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:13.177281 (+ 4697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:13.177285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:13.177312 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:13.177718 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:13.177730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.179143 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.179151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:13.180848 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:13.180861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.183382 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.183390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:13.183599 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.183632 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:13:13.183711 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:13:13.183830 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"mutex_wait_us":4,"thread_start_us":1156,"threads_started":1}
I20260924 19:13:13.185364 18560 tablet_service.cc:1467] Processing CreateTablet for tablet 16585373e3044b42b5d546c3d4d3f6f1 (DEFAULT_TABLE table=test-table [id=9a354661f0b244b195feb6e906e76111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:13.185765 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16585373e3044b42b5d546c3d4d3f6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:13.186514 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.170791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.11.18.67:37589)
0924 19:13:13.176965 (+ 6174us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:13.176971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:13.176987 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:13:13.177044 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:13.177047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:13.177104 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:13.177250 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:13.177888 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.179000 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.181161 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.182987 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.185789 (+ 2802us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.185918 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:13.185979 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:13:13.186130 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6004,"thread_start_us":5885,"threads_started":1}
I20260924 19:13:13.187294 18701 tablet_service.cc:1467] Processing CreateTablet for tablet 16585373e3044b42b5d546c3d4d3f6f1 (DEFAULT_TABLE table=test-table [id=9a354661f0b244b195feb6e906e76111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:13.187721 18701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16585373e3044b42b5d546c3d4d3f6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:13.192423 18874 tablet_bootstrap.cc:492] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: Bootstrap starting.
I20260924 19:13:13.193858 18875 tablet_bootstrap.cc:492] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: Bootstrap starting.
I20260924 19:13:13.194742 18874 tablet_bootstrap.cc:654] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:13.195097 18876 tablet_bootstrap.cc:492] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: Bootstrap starting.
I20260924 19:13:13.195556 18875 tablet_bootstrap.cc:654] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:13.196939 18876 tablet_bootstrap.cc:654] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:13.197458 18874 tablet_bootstrap.cc:492] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: No bootstrap required, opened a new log
I20260924 19:13:13.197615 18874 ts_tablet_manager.cc:1397] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 19:13:13.197938 18875 tablet_bootstrap.cc:492] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: No bootstrap required, opened a new log
I20260924 19:13:13.198203 18875 ts_tablet_manager.cc:1397] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260924 19:13:13.198659 18874 raft_consensus.cc:348] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.198935 18874 raft_consensus.cc:374] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:13.199117 18874 raft_consensus.cc:729] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0d01f02d064ea0bbb746e824d86c52, State: Initialized, Role: FOLLOWER
I20260924 19:13:13.199119 18875 raft_consensus.cc:348] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.199452 18875 raft_consensus.cc:374] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:13.199555 18875 raft_consensus.cc:729] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8264c37ea70a460e94c0be82040316c4, State: Initialized, Role: FOLLOWER
I20260924 19:13:13.199412 18874 consensus_queue.cc:260] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.199728 18876 tablet_bootstrap.cc:492] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: No bootstrap required, opened a new log
I20260924 19:13:13.199889 18876 ts_tablet_manager.cc:1397] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 19:13:13.199828 18875 consensus_queue.cc:260] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.200714 18876 raft_consensus.cc:348] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.201007 18876 raft_consensus.cc:374] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:13.201061 18875 ts_tablet_manager.cc:1428] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 19:13:13.201145 18876 raft_consensus.cc:729] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c8923bc3f947d79e290801c4f0ee8c, State: Initialized, Role: FOLLOWER
I20260924 19:13:13.201464 18876 consensus_queue.cc:260] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.202325 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.201899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:49899)
0924 19:13:13.202067 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.202160 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:13:13.205545 18478 heartbeater.cc:503] Master 127.11.18.125:35333 was elected leader, sending a full tablet report...
I20260924 19:13:13.205796 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.205077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:53499)
0924 19:13:13.205406 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.205517 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 19:13:13.207767 18876 ts_tablet_manager.cc:1428] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260924 19:13:13.208954 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.208487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:48045)
0924 19:13:13.208675 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.208784 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 19:13:13.209311 18874 ts_tablet_manager.cc:1428] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: Time spent starting tablet: real 0.012s user 0.002s sys 0.007s
I20260924 19:13:13.210556 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.210029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:50653)
0924 19:13:13.210253 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.210364 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 19:13:13.210613 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.209975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:44291)
0924 19:13:13.210364 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.210457 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
I20260924 19:13:13.210906 18768 heartbeater.cc:503] Master 127.11.18.125:35333 was elected leader, sending a full tablet report...
I20260924 19:13:13.212785 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.212409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:34783)
0924 19:13:13.212535 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.212624 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:13:13.213958 18627 heartbeater.cc:503] Master 127.11.18.125:35333 was elected leader, sending a full tablet report...
I20260924 19:13:13.297514 18881 raft_consensus.cc:482] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:13.297755 18881 raft_consensus.cc:504] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.298487 18881 leader_election.cc:288] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589), bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157)
I20260924 19:13:13.299239 18721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16585373e3044b42b5d546c3d4d3f6f1" candidate_uuid: "8264c37ea70a460e94c0be82040316c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c8923bc3f947d79e290801c4f0ee8c" is_pre_election: true
I20260924 19:13:13.299304 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16585373e3044b42b5d546c3d4d3f6f1" candidate_uuid: "8264c37ea70a460e94c0be82040316c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d01f02d064ea0bbb746e824d86c52" is_pre_election: true
I20260924 19:13:13.299507 18721 raft_consensus.cc:2454] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8264c37ea70a460e94c0be82040316c4 in term 0.
I20260924 19:13:13.299528 18580 raft_consensus.cc:2454] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8264c37ea70a460e94c0be82040316c4 in term 0.
I20260924 19:13:13.300035 18367 leader_election.cc:302] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c8923bc3f947d79e290801c4f0ee8c, 8264c37ea70a460e94c0be82040316c4; no voters:
I20260924 19:13:13.300323 18881 raft_consensus.cc:2781] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:13.300447 18881 raft_consensus.cc:482] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:13.300554 18881 raft_consensus.cc:3037] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:13.302780 18881 raft_consensus.cc:504] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.303308 18881 leader_election.cc:288] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [CANDIDATE]: Term 1 election: Requested vote from peers 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589), bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157)
I20260924 19:13:13.303814 18721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16585373e3044b42b5d546c3d4d3f6f1" candidate_uuid: "8264c37ea70a460e94c0be82040316c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c8923bc3f947d79e290801c4f0ee8c"
I20260924 19:13:13.303866 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16585373e3044b42b5d546c3d4d3f6f1" candidate_uuid: "8264c37ea70a460e94c0be82040316c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d01f02d064ea0bbb746e824d86c52"
I20260924 19:13:13.304024 18721 raft_consensus.cc:3037] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:13.304075 18580 raft_consensus.cc:3037] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:13.306221 18721 raft_consensus.cc:2454] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8264c37ea70a460e94c0be82040316c4 in term 1.
I20260924 19:13:13.306243 18580 raft_consensus.cc:2454] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8264c37ea70a460e94c0be82040316c4 in term 1.
I20260924 19:13:13.306630 18367 leader_election.cc:302] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c8923bc3f947d79e290801c4f0ee8c, 8264c37ea70a460e94c0be82040316c4; no voters:
I20260924 19:13:13.306964 18881 raft_consensus.cc:2781] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:13.307430 18881 raft_consensus.cc:686] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 1 LEADER]: Becoming Leader. State: Replica: 8264c37ea70a460e94c0be82040316c4, State: Running, Role: LEADER
I20260924 19:13:13.307808 18881 consensus_queue.cc:237] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } }
I20260924 19:13:13.308477 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.308032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:38263)
0924 19:13:13.308255 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.308352 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 19:13:13.311172 18107 catalog_manager.cc:5495] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8264c37ea70a460e94c0be82040316c4 (127.11.18.65). New cstate: current_term: 1 leader_uuid: "8264c37ea70a460e94c0be82040316c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8264c37ea70a460e94c0be82040316c4" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 } health_report { overall_health: UNKNOWN } } }
I20260924 19:13:13.333195 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.328438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.11.18.65:37241)
0924 19:13:13.328629 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:13.328634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:13.328698 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:13:13.328944 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:13.328949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:13.329014 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:13.329092 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:13.329558 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.330269 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.331699 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.332093 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.332184 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.332574 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:13.332666 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:13.332732 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:13.332941 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 19:13:13.333002 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:13:13.336351 18580 raft_consensus.cc:1261] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 1 FOLLOWER]: Refusing update from remote peer 8264c37ea70a460e94c0be82040316c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:13.336421 18721 raft_consensus.cc:1261] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 1 FOLLOWER]: Refusing update from remote peer 8264c37ea70a460e94c0be82040316c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:13.337157 18883 consensus_queue.cc:1035] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c8923bc3f947d79e290801c4f0ee8c" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:13.337671 18881 consensus_queue.cc:1035] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0d01f02d064ea0bbb746e824d86c52" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:13.340806 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.340271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:42121)
0924 19:13:13.340529 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.340607 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 19:13:13.341027 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.340375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:57671)
0924 19:13:13.340708 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.340821 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260924 19:13:13.348443 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.347346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:35545)
0924 19:13:13.348169 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.348272 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675}
I20260924 19:13:13.349444 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.348928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.67:33067)
0924 19:13:13.349126 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.349217 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 19:13:13.353006 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.352489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.66:36003)
0924 19:13:13.352728 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.352817 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 19:13:13.353964 18108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:13.354174 18108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:13.357712 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.356515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.11.18.65:38023)
0924 19:13:13.357469 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.357561 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":781}
I20260924 19:13:13.360351 18108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddd1d7a338bb4df4b3777224dafcfd12: Sending DeleteTablet for 3 replicas of tablet 16585373e3044b42b5d546c3d4d3f6f1
I20260924 19:13:13.361255 18411 tablet_service.cc:1514] Processing DeleteTablet for tablet 16585373e3044b42b5d546c3d4d3f6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:13 UTC) from {username='oryx', principal='oryx/127.11.18.125@KRBTEST.COM'} at 127.0.0.1:49722
I20260924 19:13:13.361615 18560 tablet_service.cc:1514] Processing DeleteTablet for tablet 16585373e3044b42b5d546c3d4d3f6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:13 UTC) from {username='oryx', principal='oryx/127.11.18.125@KRBTEST.COM'} at 127.0.0.1:32910
I20260924 19:13:13.361989 18701 tablet_service.cc:1514] Processing DeleteTablet for tablet 16585373e3044b42b5d546c3d4d3f6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:13 UTC) from {username='oryx', principal='oryx/127.11.18.125@KRBTEST.COM'} at 127.0.0.1:45852
I20260924 19:13:13.362006 18559 tablet_service.cc:1514] Processing DeleteTablet for tablet 16585373e3044b42b5d546c3d4d3f6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:13 UTC) from {username='oryx', principal='oryx/127.11.18.125@KRBTEST.COM'} at 127.0.0.1:32910
I20260924 19:13:13.363572 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 18277
I20260924 19:13:13.373112 18897 tablet_replica.cc:330] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: stopping tablet replica
I20260924 19:13:13.373456 18897 raft_consensus.cc:2229] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:13.373855 18897 raft_consensus.cc:2258] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:13.374373 18895 tablet_replica.cc:330] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: stopping tablet replica
I20260924 19:13:13.374465 18896 tablet_replica.cc:330] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: stopping tablet replica
I20260924 19:13:13.374655 18896 raft_consensus.cc:2229] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:13.374810 18895 raft_consensus.cc:2229] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:13:13.374982 18896 raft_consensus.cc:2258] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:13.375248 18895 raft_consensus.cc:2258] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:13.376585 18895 ts_tablet_manager.cc:1905] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:13.377413 18896 ts_tablet_manager.cc:1905] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:13.377703 18897 ts_tablet_manager.cc:1905] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:13.383764 18895 ts_tablet_manager.cc:1918] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:13.384042 18895 log.cc:1196] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/16585373e3044b42b5d546c3d4d3f6f1
I20260924 19:13:13.384228 18896 ts_tablet_manager.cc:1918] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:13.384390 18896 log.cc:1196] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/16585373e3044b42b5d546c3d4d3f6f1
I20260924 19:13:13.384783 18895 ts_tablet_manager.cc:1939] T 16585373e3044b42b5d546c3d4d3f6f1 P 8264c37ea70a460e94c0be82040316c4: Deleting consensus metadata
I20260924 19:13:13.384915 18896 ts_tablet_manager.cc:1939] T 16585373e3044b42b5d546c3d4d3f6f1 P bb0d01f02d064ea0bbb746e824d86c52: Deleting consensus metadata
I20260924 19:13:13.385443 18897 ts_tablet_manager.cc:1918] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:13.385681 18897 log.cc:1196] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/16585373e3044b42b5d546c3d4d3f6f1
I20260924 19:13:13.386096 18092 catalog_manager.cc:4841] TS 8264c37ea70a460e94c0be82040316c4 (127.11.18.65:37241): tablet 16585373e3044b42b5d546c3d4d3f6f1 (table test-table [id=9a354661f0b244b195feb6e906e76111]) successfully deleted
I20260924 19:13:13.386108 18095 catalog_manager.cc:4841] TS bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157): tablet 16585373e3044b42b5d546c3d4d3f6f1 (table test-table [id=9a354661f0b244b195feb6e906e76111]) successfully deleted
I20260924 19:13:13.386531 18897 ts_tablet_manager.cc:1939] T 16585373e3044b42b5d546c3d4d3f6f1 P 80c8923bc3f947d79e290801c4f0ee8c: Deleting consensus metadata
W20260924 19:13:13.386798 18095 catalog_manager.cc:4808] TS bb0d01f02d064ea0bbb746e824d86c52 (127.11.18.66:40157): delete failed for tablet 16585373e3044b42b5d546c3d4d3f6f1 because the tablet was not found. No further retry: Not found: Tablet not found: 16585373e3044b42b5d546c3d4d3f6f1
I20260924 19:13:13.387739 18095 catalog_manager.cc:4841] TS 80c8923bc3f947d79e290801c4f0ee8c (127.11.18.67:37589): tablet 16585373e3044b42b5d546c3d4d3f6f1 (table test-table [id=9a354661f0b244b195feb6e906e76111]) successfully deleted
I20260924 19:13:13.393980 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.364864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.11.18.65:37241)
0924 19:13:13.365094 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:13.365104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:13.375690 (+ 10586us) server_negotiation.cc:407] Connection header received
0924 19:13:13.375945 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:13.375950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:13.376057 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:13.376196 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:13.378805 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.380039 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.382144 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.382897 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.383025 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.387413 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:13.387428 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:13.387443 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:13.387481 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:13.391051 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:13.391785 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:13.391795 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:13.391799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:13.391952 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:13.392374 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:13.392379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:13.392383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:13.392781 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:13.393075 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:13.393413 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 19:13:13.393628 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:13:13.394622 18471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:13.515890 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 18277
I20260924 19:13:13.544950 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 18488
I20260924 19:13:13.559660 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.545733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.11.18.66:40157)
0924 19:13:13.545932 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:13.545936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:13.546015 (+ 79us) server_negotiation.cc:407] Connection header received
0924 19:13:13.546198 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:13.546202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:13.546268 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:13.546386 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:13.548085 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.548829 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.550357 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.550868 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.550985 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.553804 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:13.553820 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:13.553838 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:13.553897 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:13.556879 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:13.557753 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:13.557760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:13.557763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:13.557929 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:13.558270 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:13.558273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:13.558275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:13.558630 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:13.558895 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:13.559213 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 19:13:13.559402 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 19:13:13.560444 18620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:13.657758 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 18488
I20260924 19:13:13.676007 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 18638
I20260924 19:13:13.690992 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.676913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.11.18.67:37589)
0924 19:13:13.677224 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:13.677232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:13.677254 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:13:13.677409 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:13.677414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:13.677487 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:13.677623 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:13.679095 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.679837 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.681063 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:13.681536 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:13.681662 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:13.684408 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:13.684424 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:13.684442 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:13.684477 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:13.687909 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:13.688710 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:13.688718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:13.688721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:13.688869 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:13.689278 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:13.689282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:13.689284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:13.689709 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:13.689958 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:13.690403 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 19:13:13.690594 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:13:13.691722 18761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:13.776276 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.775243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48796)
0924 19:13:13.775983 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.776078 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":492,"threads_started":1}
I20260924 19:13:13.777730 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:13.777295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34731 (local address 127.0.0.1:48798)
0924 19:13:13.777521 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:13.777602 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:34731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 19:13:13.787356 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 18638
I20260924 19:13:13.805023 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 17971
I20260924 19:13:13.806278 18138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:13.928375 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 17971
I20260924 19:13:13.950548 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 18070
I20260924 19:13:13.952071 18328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:14.067199 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 18070
2026-09-24T19:13:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11872 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:13:14 dist-test-slave-0lgr krb5kdc[18908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:13:14 dist-test-slave-0lgr krb5kdc[18908](info): set up 2 sockets
Sep 24 19:13:14 dist-test-slave-0lgr krb5kdc[18908](info): commencing operation
krb5kdc: starting...
W20260924 19:13:15.226965 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:13:15 dist-test-slave-0lgr krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:13:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:13:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.126.keytab.
Entry for principal oryx/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.126.keytab.
I20260924 19:13:15.462914 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36381
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.126.keytab
--principal=oryx/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:36381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:15.819485 18921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:15.819816 18921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:15.819990 18921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:15.828315 18921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:15.828434 18921 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:13:15.828524 18921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:15.828627 18921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:15.828725 18921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:15.828846 18921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:15.839072 18921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:36381
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.126.keytab
--principal=oryx/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:15.840577 18921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:15.841971 18921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:13:15.854589 18921 server_base.cc:1034] running on GCE node
W20260924 19:13:15.854540 18927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:15.854604 18926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:15.854930 18929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:15.855836 18921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:15.857291 18921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:15.858671 18921 hybrid_clock.cc:648] HybridClock initialized: now 1790277195858601 us; error 39 us; skew 500 ppm
I20260924 19:13:15.859040 18921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:15 dist-test-slave-0lgr krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277195, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:15.862703 18921 init.cc:377] Logged in from keytab as oryx/127.11.18.126@KRBTEST.COM (short username oryx)
I20260924 19:13:15.864308 18921 webserver.cc:460] Webserver started at http://127.11.18.126:33029/ using document root <none> and password file <none>
I20260924 19:13:15.865132 18921 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:15.865273 18921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:15.865656 18921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:15.868820 18921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "fc8f661014fe455bb862a5ced5156261"
format_stamp: "Formatted at 2026-09-24 19:13:15 on dist-test-slave-0lgr"
server_key: "bb53d20fd5aa54c002a6a38f76bbb6fa"
server_key_iv: "ae56acd704909e591e7b2906fc4ffafc"
server_key_version: "encryptionkey@0"
I20260924 19:13:15.869763 18921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "fc8f661014fe455bb862a5ced5156261"
format_stamp: "Formatted at 2026-09-24 19:13:15 on dist-test-slave-0lgr"
server_key: "bb53d20fd5aa54c002a6a38f76bbb6fa"
server_key_iv: "ae56acd704909e591e7b2906fc4ffafc"
server_key_version: "encryptionkey@0"
I20260924 19:13:15.875119 18921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:13:15.878695 18936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:15.880112 18921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:15.880354 18921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "fc8f661014fe455bb862a5ced5156261"
format_stamp: "Formatted at 2026-09-24 19:13:15 on dist-test-slave-0lgr"
server_key: "bb53d20fd5aa54c002a6a38f76bbb6fa"
server_key_iv: "ae56acd704909e591e7b2906fc4ffafc"
server_key_version: "encryptionkey@0"
I20260924 19:13:15.880579 18921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:15.897068 18921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:15.901594 18921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:15.901886 18921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:15.924973 18921 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:36381
I20260924 19:13:15.924993 18997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:36381 every 8 connection(s)
I20260924 19:13:15.926419 18921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:13:15.930364 18998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:15.936103 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 18921
I20260924 19:13:15.936307 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:13:15.937179 11337 external_mini_cluster.cc:1428] Setting key 9179f825ff807eea288c89a55c919cd0
I20260924 19:13:15.942265 18998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: Bootstrap starting.
I20260924 19:13:15.946101 18998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:15.947400 18998 log.cc:824] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:15.951054 18998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: No bootstrap required, opened a new log
Sep 24 19:13:15 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:15.959369 18998 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8f661014fe455bb862a5ced5156261" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36381 } }
I20260924 19:13:15.959790 18998 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:15.959914 18998 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8f661014fe455bb862a5ced5156261, State: Initialized, Role: FOLLOWER
I20260924 19:13:15.960640 18998 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8f661014fe455bb862a5ced5156261" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36381 } }
I20260924 19:13:15.960924 18998 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:13:15.961086 18998 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:13:15.961294 18998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:15.962030 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:15.939484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.11.18.126:36381)
0924 19:13:15.940380 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:15.940392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:15.940433 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:13:15.942232 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:15.942264 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:15.942695 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:15.943024 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:15.944943 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:15.946446 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:15.948346 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:15.949177 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:15.949473 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:15.954146 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:15.954155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:15.954174 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:15.954219 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:15.957465 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:15.958185 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:15.958200 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:15.958207 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:15.958385 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:15.958806 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:15.958815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:15.958818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:15.959377 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:15.959689 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:15.960125 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 19:13:15.960667 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":443,"threads_started":1}
I20260924 19:13:15.963765 18998 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8f661014fe455bb862a5ced5156261" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36381 } }
I20260924 19:13:15.964320 18998 leader_election.cc:302] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc8f661014fe455bb862a5ced5156261; no voters:
I20260924 19:13:15.964982 18998 leader_election.cc:288] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:13:15.965106 19003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:15.965883 19003 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [term 1 LEADER]: Becoming Leader. State: Replica: fc8f661014fe455bb862a5ced5156261, State: Running, Role: LEADER
I20260924 19:13:15.966444 19003 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8f661014fe455bb862a5ced5156261" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36381 } }
I20260924 19:13:15.967264 18998 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:15.969630 19005 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc8f661014fe455bb862a5ced5156261. Latest consensus state: current_term: 1 leader_uuid: "fc8f661014fe455bb862a5ced5156261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8f661014fe455bb862a5ced5156261" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36381 } } }
I20260924 19:13:15.969578 19004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc8f661014fe455bb862a5ced5156261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8f661014fe455bb862a5ced5156261" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36381 } } }
I20260924 19:13:15.970182 19004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:15.970048 19005 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:15.974792 19009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:13:15.980626 19009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:13:15.989348 19009 catalog_manager.cc:1347] Generated new cluster ID: 5d48fa17cb584bce80e648f2a1fce276
I20260924 19:13:15.989468 19009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:13:16.005419 19009 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:13:16.007346 19009 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:13:16.021412 19009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: Generated new TSK 0
I20260924 19:13:16.022328 19009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab.
Entry for principal oryx/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab.
I20260924 19:13:16.086047 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36381
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab
--principal=oryx/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:16.423621 19024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:16.423930 19024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:16.424109 19024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:16.431383 19024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:16.431504 19024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:16.431648 19024 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:13:16.431797 19024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:16.441766 19024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab
--principal=oryx/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:36381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:16.443092 19024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:16.444428 19024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:16.457572 19030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:16.457608 19032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:16.458117 19029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:16.458405 19024 server_base.cc:1034] running on GCE node
I20260924 19:13:16.458959 19024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:16.459854 19024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:16.461103 19024 hybrid_clock.cc:648] HybridClock initialized: now 1790277196461038 us; error 54 us; skew 500 ppm
I20260924 19:13:16.461477 19024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:16 dist-test-slave-0lgr krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277196, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:16.464644 19024 init.cc:377] Logged in from keytab as oryx/127.11.18.65@KRBTEST.COM (short username oryx)
I20260924 19:13:16.466159 19024 webserver.cc:460] Webserver started at http://127.11.18.65:33785/ using document root <none> and password file <none>
I20260924 19:13:16.466812 19024 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:16.466899 19024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:16.467149 19024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:16.470172 19024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "f5d4d3c6eef04c0c8223d710d363fd2e"
format_stamp: "Formatted at 2026-09-24 19:13:16 on dist-test-slave-0lgr"
server_key: "3f29b414d0c9aab69da7a01c5bca09ba"
server_key_iv: "87401b4eabe0cbba89e8f20da7bcf10f"
server_key_version: "encryptionkey@0"
I20260924 19:13:16.470890 19024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "f5d4d3c6eef04c0c8223d710d363fd2e"
format_stamp: "Formatted at 2026-09-24 19:13:16 on dist-test-slave-0lgr"
server_key: "3f29b414d0c9aab69da7a01c5bca09ba"
server_key_iv: "87401b4eabe0cbba89e8f20da7bcf10f"
server_key_version: "encryptionkey@0"
I20260924 19:13:16.476445 19024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:13:16.479924 19039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:16.481258 19024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:13:16.481513 19024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "f5d4d3c6eef04c0c8223d710d363fd2e"
format_stamp: "Formatted at 2026-09-24 19:13:16 on dist-test-slave-0lgr"
server_key: "3f29b414d0c9aab69da7a01c5bca09ba"
server_key_iv: "87401b4eabe0cbba89e8f20da7bcf10f"
server_key_version: "encryptionkey@0"
I20260924 19:13:16.481751 19024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:16.500370 19024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:16.504855 19024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:16.505148 19024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:16.507957 19024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:16.508074 19024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:16.508272 19024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:16.508410 19024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 19:13:16 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277196, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:16.531009 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:16.508499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:50209 (local address 127.11.18.126:36381)
0924 19:13:16.509458 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:16.509464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:16.510445 (+ 981us) server_negotiation.cc:407] Connection header received
0924 19:13:16.512606 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:16.512612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:16.512706 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:16.512833 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:16.515230 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:16.516320 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:16.519145 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:16.519704 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:16.519854 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:16.524134 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:16.524139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:16.524143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:16.524175 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:16.527222 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:16.528419 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:16.528428 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:16.528434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:16.528622 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:16.529151 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:16.529159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:16.529162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:16.529523 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:16.529681 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:16.530429 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 19:13:16.530655 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":596,"threads_started":1}
I20260924 19:13:16.532099 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:16.509184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36381 (local address 127.11.18.65:50209)
0924 19:13:16.510136 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:16.510194 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:16.512242 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:16.513099 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:16.513115 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:16.514064 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:16.515014 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:16.515047 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:16.516520 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:16.516527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:16.518973 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:16.518986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:16.519974 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:16.519986 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:16.520227 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:16.521254 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:16.521282 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:16.523877 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:16.527394 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:16.527402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:16.527417 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:16.528171 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:16.528759 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:16.528766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:16.528773 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:16.529011 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:16.529655 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:16.529663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:16.530055 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 19:13:16.530324 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 19:13:16.530876 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":370,"threads_started":1}
I20260924 19:13:16.556512 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:16.539526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:55109 (local address 127.11.18.126:36381)
0924 19:13:16.539682 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:16.539686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:16.540507 (+ 821us) server_negotiation.cc:407] Connection header received
0924 19:13:16.540709 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:16.540714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:16.540811 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:16.540921 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:16.542182 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:16.543235 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:16.545303 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:16.545919 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:16.546063 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:16.548914 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:16.548924 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:16.548932 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:16.548984 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:16.553188 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:16.554236 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:16.554241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:16.554243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:16.554381 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:16.554865 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:16.554870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:16.554875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:16.555236 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:16.555412 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:16.555888 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 19:13:16.556133 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:13:16.556730 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:16.539640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36381 (local address 127.11.18.65:55109)
0924 19:13:16.540382 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:16.540400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:16.540583 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:16.540946 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:16.540950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:16.541512 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:16.541955 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:16.541975 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:16.543397 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:16.543412 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:16.545123 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:16.545135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:16.546396 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:16.546401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:16.546620 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:16.547511 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:16.547540 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:16.548678 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:16.553452 (+ 4774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:16.553458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:16.553463 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:16.554079 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:16.554504 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:16.554508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:16.554511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:16.554687 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:16.555385 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:16.555391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:16.555584 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 19:13:16.555750 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 19:13:16.556245 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":390,"threads_started":1}
I20260924 19:13:16.565831 19024 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:42201
I20260924 19:13:16.565871 19159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:42201 every 8 connection(s)
I20260924 19:13:16.567034 19024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:16.569280 19160 heartbeater.cc:347] Connected to a master server at 127.11.18.126:36381
I20260924 19:13:16.569584 19160 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:16.569806 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 19024
I20260924 19:13:16.570004 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:13:16.570438 19160 heartbeater.cc:511] Master 127.11.18.126:36381 requested a full tablet report, sending...
I20260924 19:13:16.570699 11337 external_mini_cluster.cc:1428] Setting key 15039e3efae3809cb78d8a3671e02390
I20260924 19:13:16.573072 18952 ts_manager.cc:194] Registered new tserver with Master: f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201)
I20260924 19:13:16.574867 18952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:50209
WARNING: no policy specified for oryx/127.11.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab.
Entry for principal oryx/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab.
I20260924 19:13:16.622570 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36381
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab
--principal=oryx/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:16.965185 19164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:16.965600 19164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:16.965801 19164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:16.973464 19164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:16.973575 19164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:16.973743 19164 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:13:16.973881 19164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:16.983919 19164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab
--principal=oryx/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:36381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:16.985215 19164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:16.986636 19164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:17.000584 19169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:17.000675 19170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:17.000584 19172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:17.001956 19164 server_base.cc:1034] running on GCE node
I20260924 19:13:17.002653 19164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:17.003712 19164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:17.004964 19164 hybrid_clock.cc:648] HybridClock initialized: now 1790277197004906 us; error 37 us; skew 500 ppm
I20260924 19:13:17.005275 19164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:17 dist-test-slave-0lgr krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277197, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:17.008536 19164 init.cc:377] Logged in from keytab as oryx/127.11.18.66@KRBTEST.COM (short username oryx)
I20260924 19:13:17.010221 19164 webserver.cc:460] Webserver started at http://127.11.18.66:38329/ using document root <none> and password file <none>
I20260924 19:13:17.010943 19164 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:17.011075 19164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:17.011396 19164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:17.014552 19164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "762cdc846a634fffa6de3d4ffbd5ee35"
format_stamp: "Formatted at 2026-09-24 19:13:17 on dist-test-slave-0lgr"
server_key: "2aaf4b2c80412ac17e52e76356117787"
server_key_iv: "60305c56d10fe636587d87eed6a6605a"
server_key_version: "encryptionkey@0"
I20260924 19:13:17.015345 19164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "762cdc846a634fffa6de3d4ffbd5ee35"
format_stamp: "Formatted at 2026-09-24 19:13:17 on dist-test-slave-0lgr"
server_key: "2aaf4b2c80412ac17e52e76356117787"
server_key_iv: "60305c56d10fe636587d87eed6a6605a"
server_key_version: "encryptionkey@0"
I20260924 19:13:17.020653 19164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:13:17.024511 19179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:17.026057 19164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:13:17.026237 19164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "762cdc846a634fffa6de3d4ffbd5ee35"
format_stamp: "Formatted at 2026-09-24 19:13:17 on dist-test-slave-0lgr"
server_key: "2aaf4b2c80412ac17e52e76356117787"
server_key_iv: "60305c56d10fe636587d87eed6a6605a"
server_key_version: "encryptionkey@0"
I20260924 19:13:17.026459 19164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:17.046324 19164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:17.050992 19164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:17.051295 19164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:17.054086 19164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:17.054265 19164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 19:13:17.054461 19164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:17.054575 19164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:13:17 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277197, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:17.081199 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:17.054711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:41019 (local address 127.11.18.126:36381)
0924 19:13:17.054968 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:17.054975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:17.056398 (+ 1423us) server_negotiation.cc:407] Connection header received
0924 19:13:17.058898 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:17.058918 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:17.059017 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:17.059131 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:17.060710 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.061821 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.064603 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.065491 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.065710 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:17.072283 (+ 6573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:17.072292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:17.072297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:17.072348 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:17.076586 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:17.077861 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:17.077870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:17.077876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:17.078049 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:17.078764 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:17.078770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:17.078773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:17.079057 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:17.079185 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:17.080457 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0924 19:13:17.080749 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:13:17.083117 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:17.055328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36381 (local address 127.11.18.66:41019)
0924 19:13:17.056187 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:17.056244 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:17.058538 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:17.059375 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:17.059384 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:17.059906 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:17.060527 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:17.060544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.062012 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.062025 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:17.064390 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:17.064412 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.065877 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.065894 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:17.066426 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:17.067460 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:17.067501 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:17.071892 (+ 4391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:17.076810 (+ 4918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:17.076826 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:17.076852 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:17.077700 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:17.078227 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:17.078235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:17.078245 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:17.078604 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:17.079214 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:17.079232 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:17.079889 (+ 657us) client_negotiation.cc:769] Sending connection context
0924 19:13:17.080281 (+ 392us) client_negotiation.cc:240] Negotiation successful
0924 19:13:17.081023 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":372,"threads_started":1}
I20260924 19:13:17.097100 19164 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:33207
I20260924 19:13:17.097137 19298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:33207 every 8 connection(s)
I20260924 19:13:17.098560 19164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:17.100965 19299 heartbeater.cc:347] Connected to a master server at 127.11.18.126:36381
I20260924 19:13:17.101706 19299 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:17.102421 19299 heartbeater.cc:511] Master 127.11.18.126:36381 requested a full tablet report, sending...
I20260924 19:13:17.103984 18952 ts_manager.cc:194] Registered new tserver with Master: 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207)
I20260924 19:13:17.104868 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:17.090575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36381 (local address 127.11.18.66:48917)
0924 19:13:17.091389 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:17.091414 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:17.091593 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:17.091960 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:17.091965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:17.092434 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:17.092732 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:17.092741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.094190 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.094199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:17.095708 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:17.095724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.096690 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.096697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:17.096894 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:17.097587 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:17.097605 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:17.098459 (+ 854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:17.102066 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:17.102072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:17.102077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:17.102494 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:17.102954 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:17.102959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:17.102961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:17.103101 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:17.103917 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:17.103920 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:17.104124 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 19:13:17.104327 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 19:13:17.104568 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":468,"threads_started":1}
I20260924 19:13:17.104974 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:17.090406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:48917 (local address 127.11.18.126:36381)
0924 19:13:17.090647 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:17.090655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:17.091517 (+ 862us) server_negotiation.cc:407] Connection header received
0924 19:13:17.091737 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:17.091743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:17.091821 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:17.091930 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:17.092882 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.093978 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.095865 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.096479 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.096636 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:17.098684 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:17.098690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:17.098694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:17.098735 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:17.101931 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:17.102625 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:17.102632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:17.102636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:17.102801 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:17.103300 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:17.103309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:17.103312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:17.103772 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:17.103935 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:17.104384 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 19:13:17.104595 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 19:13:17.105083 18952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:41019
I20260924 19:13:17.105592 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 19164
I20260924 19:13:17.105757 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:13:17.106381 11337 external_mini_cluster.cc:1428] Setting key 00856106aa6b00eb5478cd497c3b5dad
WARNING: no policy specified for oryx/127.11.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab.
Entry for principal oryx/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab.
I20260924 19:13:17.157979 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36381
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab
--principal=oryx/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:17.487617 19303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:17.487911 19303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:17.488008 19303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:17.495225 19303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:17.495353 19303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:17.495437 19303 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:13:17.495522 19303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:17.505376 19303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab
--principal=oryx/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.18.126:36381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:17.506642 19303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:17.508016 19303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:17.521757 19309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:17.521814 19308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:17.521731 19311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:17.522907 19303 server_base.cc:1034] running on GCE node
I20260924 19:13:17.523625 19303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:17.524641 19303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:17.525910 19303 hybrid_clock.cc:648] HybridClock initialized: now 1790277197525826 us; error 53 us; skew 500 ppm
I20260924 19:13:17.526332 19303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:17 dist-test-slave-0lgr krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277197, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:17.530186 19303 init.cc:377] Logged in from keytab as oryx/127.11.18.67@KRBTEST.COM (short username oryx)
I20260924 19:13:17.531919 19303 webserver.cc:460] Webserver started at http://127.11.18.67:38949/ using document root <none> and password file <none>
I20260924 19:13:17.532631 19303 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:17.532766 19303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:17.533092 19303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:17.536442 19303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "b8883cb17cd24f379f4e8298df93d088"
format_stamp: "Formatted at 2026-09-24 19:13:17 on dist-test-slave-0lgr"
server_key: "d53741dcb1a3a92f59e507ce082150d9"
server_key_iv: "fc0794f082461c6ae6d5418de3e7999f"
server_key_version: "encryptionkey@0"
I20260924 19:13:17.537308 19303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "b8883cb17cd24f379f4e8298df93d088"
format_stamp: "Formatted at 2026-09-24 19:13:17 on dist-test-slave-0lgr"
server_key: "d53741dcb1a3a92f59e507ce082150d9"
server_key_iv: "fc0794f082461c6ae6d5418de3e7999f"
server_key_version: "encryptionkey@0"
I20260924 19:13:17.543157 19303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:13:17.546918 19318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:17.548506 19303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:17.548741 19303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "b8883cb17cd24f379f4e8298df93d088"
format_stamp: "Formatted at 2026-09-24 19:13:17 on dist-test-slave-0lgr"
server_key: "d53741dcb1a3a92f59e507ce082150d9"
server_key_iv: "fc0794f082461c6ae6d5418de3e7999f"
server_key_version: "encryptionkey@0"
I20260924 19:13:17.549001 19303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:17.574712 19303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:17.578229 19160 heartbeater.cc:503] Master 127.11.18.126:36381 was elected leader, sending a full tablet report...
I20260924 19:13:17.580289 19303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:17.580570 19303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:17.583348 19303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:17.583437 19303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:17.583649 19303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:17.583741 19303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:13:17 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277197, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:17.611901 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:17.583982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:55641 (local address 127.11.18.126:36381)
0924 19:13:17.584288 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:17.584296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:17.585954 (+ 1658us) server_negotiation.cc:407] Connection header received
0924 19:13:17.588253 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:17.588261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:17.588386 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:17.588518 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:17.590403 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.591684 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.595313 (+ 3629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.596013 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.596152 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:17.603682 (+ 7530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:17.603691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:17.603697 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:17.603753 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:17.607216 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:17.608525 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:17.608530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:17.608532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:17.608659 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:17.609470 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:17.609475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:17.609478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:17.609884 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:17.610059 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:17.611397 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0924 19:13:17.611604 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 19:13:17.614372 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:17.584695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36381 (local address 127.11.18.67:55641)
0924 19:13:17.585792 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:17.585844 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:17.587892 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:17.588808 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:17.588818 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:17.589670 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:17.590228 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:17.590245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.591879 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.591886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:17.595056 (+ 3170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:17.595082 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.596955 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.596969 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:17.597436 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:17.598590 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:17.598642 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:17.603278 (+ 4636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:17.607408 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:17.607426 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:17.607454 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:17.608352 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:17.608782 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:17.608791 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:17.608797 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:17.609283 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:17.610130 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:17.610139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:17.610752 (+ 613us) client_negotiation.cc:769] Sending connection context
0924 19:13:17.611209 (+ 457us) client_negotiation.cc:240] Negotiation successful
0924 19:13:17.611950 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":425,"threads_started":1}
I20260924 19:13:17.633638 19303 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:46411
I20260924 19:13:17.633693 19437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:46411 every 8 connection(s)
I20260924 19:13:17.635202 19303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:17.637602 19438 heartbeater.cc:347] Connected to a master server at 127.11.18.126:36381
I20260924 19:13:17.637845 19438 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:17.638537 19438 heartbeater.cc:511] Master 127.11.18.126:36381 requested a full tablet report, sending...
I20260924 19:13:17.640336 18953 ts_manager.cc:194] Registered new tserver with Master: b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411)
I20260924 19:13:17.640527 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 19303
I20260924 19:13:17.640740 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:13:17.641412 11337 external_mini_cluster.cc:1428] Setting key ff1d6bf69b89830573cf2de4220b7af3
I20260924 19:13:17.641795 18953 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:55641
I20260924 19:13:17.644865 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:17.625935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36381 (local address 127.11.18.67:50981)
0924 19:13:17.626857 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:17.626876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:17.627067 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:17.627698 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:17.627702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:17.628499 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:17.629228 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:17.629243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.630974 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.630983 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:17.632533 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:17.632547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.633759 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.633768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:17.634012 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:17.634791 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:17.634811 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:17.636020 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:17.641969 (+ 5949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:17.641976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:17.641981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:17.642522 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:17.642945 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:17.642951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:17.642955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:17.643104 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:17.643772 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:17.643778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:17.643956 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 19:13:17.644093 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 19:13:17.644423 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":434,"threads_started":1}
I20260924 19:13:17.644881 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:13:17.645843 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:17.625891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:50981 (local address 127.11.18.126:36381)
0924 19:13:17.626134 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:17.626140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:17.626986 (+ 846us) server_negotiation.cc:407] Connection header received
0924 19:13:17.627233 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:17.627261 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:17.627384 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:17.627500 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:17.629495 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.630785 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.632706 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.633526 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.633693 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:17.636304 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:17.636314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:17.636322 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:17.636374 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:17.641818 (+ 5444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:17.642663 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:17.642668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:17.642672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:17.642848 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:17.643235 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:17.643240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:17.643244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:17.643622 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:17.645261 (+ 1639us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:17.645433 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 19:13:17.645614 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Sep 24 19:13:17 dist-test-slave-0lgr krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:13:17 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:17 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:18.108945 19299 heartbeater.cc:503] Master 127.11.18.126:36381 was elected leader, sending a full tablet report...
I20260924 19:13:18.645684 19438 heartbeater.cc:503] Master 127.11.18.126:36381 was elected leader, sending a full tablet report...
W20260924 19:13:18.834199 19446 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.141s user 0.000s sys 0.015s
W20260924 19:13:18.835493 19048 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:17.686351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.11.18.126:36381)
0924 19:13:17.686592 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:17.686599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:17.687039 (+ 440us) server_negotiation.cc:407] Connection header received
0924 19:13:17.687273 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:17.687279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:17.687363 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:17.687478 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:17.688988 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.689899 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.691376 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:17.691821 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:17.691966 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:18.835167 (+1143201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45542: BlockingRecv error: recv got EOF from 127.0.0.1:45542 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 19:13:18 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:18.857455 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:18.839591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.11.18.126:36381)
0924 19:13:18.839801 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:18.839806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:18.840249 (+ 443us) server_negotiation.cc:407] Connection header received
0924 19:13:18.840401 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:18.840405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:18.840499 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:18.840599 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:18.841942 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.842661 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.843863 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.844270 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.844362 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:18.851618 (+ 7256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:18.851624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:18.851627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:18.851667 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:18.854561 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:18.855373 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:18.855377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:18.855380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:18.855504 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:18.855965 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:18.855969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:18.855972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:18.856309 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:18.856562 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:18.856842 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 19:13:18.857037 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:13:18.861696 18953 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:13:18.868626 18953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:13:18.902527 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:18.883089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.11.18.66:33207)
0924 19:13:18.894175 (+ 11086us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:18.894187 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:18.894222 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:13:18.894327 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:18.894335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:18.894597 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:18.894830 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:18.895713 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.897353 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.900168 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.901232 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.901439 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:18.901687 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:18.901975 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 19:13:18.902162 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10883,"thread_start_us":10735,"threads_started":1}
I20260924 19:13:18.903019 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:18.882578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46411 (local address 127.0.0.1:51640)
0924 19:13:18.884430 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:18.884447 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:18.884544 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:18.895393 (+ 10849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:18.895402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:18.895438 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:18.896080 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:18.896092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.897851 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.897856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:18.900242 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:18.900256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.902177 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.902184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:18.902354 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:18.902388 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:13:18.902452 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:13:18.902595 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"mutex_wait_us":110,"thread_start_us":1420,"threads_started":1}
I20260924 19:13:18.902659 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:18.882568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:33207 (local address 127.0.0.1:50096)
0924 19:13:18.883440 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:18.883488 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:18.883702 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:18.894846 (+ 11144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:18.894855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:18.894887 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:18.895442 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:18.895456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.897500 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.897508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:18.899983 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:18.899999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.901514 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.901520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:18.901716 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:18.901773 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 19:13:18.901927 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 19:13:18.902166 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":1,"thread_start_us":683,"threads_started":1}
I20260924 19:13:18.903482 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:18.882760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.11.18.67:46411)
0924 19:13:18.894587 (+ 11827us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:18.894614 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:18.894656 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:13:18.894791 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:18.894799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:18.895100 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:18.895581 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:18.896264 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.897696 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.900430 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.901937 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.902762 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:18.902953 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:18.903031 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 19:13:18.903185 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11611,"thread_start_us":11428,"threads_started":1}
I20260924 19:13:18.904518 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:18.883102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:42201 (local address 127.0.0.1:40908)
0924 19:13:18.895506 (+ 12404us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:18.895528 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:18.895757 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:18.896563 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:18.896568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:18.896595 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:18.897052 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:18.897062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.899164 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.899169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:18.901394 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:18.901408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.903452 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.903458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:18.903648 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:18.903678 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:13:18.904030 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 19:13:18.904152 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12274,"mutex_wait_us":39,"thread_start_us":12112,"threads_started":1}
I20260924 19:13:18.905305 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:18.883696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.11.18.65:42201)
0924 19:13:18.893669 (+ 9973us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:18.893679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:18.895673 (+ 1994us) server_negotiation.cc:407] Connection header received
0924 19:13:18.895975 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:18.895982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:18.896154 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:18.896563 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:18.897233 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.899005 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.901582 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:18.902990 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:18.903122 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:18.903311 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:18.904867 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0924 19:13:18.905030 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9793,"thread_start_us":9676,"threads_started":1}
I20260924 19:13:18.907358 19229 tablet_service.cc:1467] Processing CreateTablet for tablet 2d59ac6378624ff59688fbfda6a5ba8e (DEFAULT_TABLE table=test-table [id=942dee30ecc44388992b1325d6a37873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:18.907377 19368 tablet_service.cc:1467] Processing CreateTablet for tablet 2d59ac6378624ff59688fbfda6a5ba8e (DEFAULT_TABLE table=test-table [id=942dee30ecc44388992b1325d6a37873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:18.908701 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d59ac6378624ff59688fbfda6a5ba8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:18.908784 19368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d59ac6378624ff59688fbfda6a5ba8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:18.909184 19094 tablet_service.cc:1467] Processing CreateTablet for tablet 2d59ac6378624ff59688fbfda6a5ba8e (DEFAULT_TABLE table=test-table [id=942dee30ecc44388992b1325d6a37873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:18.910552 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d59ac6378624ff59688fbfda6a5ba8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:18.923229 19463 tablet_bootstrap.cc:492] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Bootstrap starting.
I20260924 19:13:18.927569 19463 tablet_bootstrap.cc:654] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:18.929036 19463 log.cc:824] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:18.931932 19464 tablet_bootstrap.cc:492] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Bootstrap starting.
I20260924 19:13:18.939741 19464 tablet_bootstrap.cc:654] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:18.941907 19464 log.cc:824] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:18.946442 19465 tablet_bootstrap.cc:492] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Bootstrap starting.
I20260924 19:13:18.948854 19463 tablet_bootstrap.cc:492] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: No bootstrap required, opened a new log
I20260924 19:13:18.949256 19463 ts_tablet_manager.cc:1397] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.005s
I20260924 19:13:18.950335 19464 tablet_bootstrap.cc:492] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: No bootstrap required, opened a new log
I20260924 19:13:18.950659 19464 ts_tablet_manager.cc:1397] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20260924 19:13:18.952412 19465 tablet_bootstrap.cc:654] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:18.953852 19465 log.cc:824] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:18.959736 19463 raft_consensus.cc:348] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:18.960175 19463 raft_consensus.cc:374] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:18.960348 19463 raft_consensus.cc:729] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762cdc846a634fffa6de3d4ffbd5ee35, State: Initialized, Role: FOLLOWER
I20260924 19:13:18.960675 19465 tablet_bootstrap.cc:492] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: No bootstrap required, opened a new log
I20260924 19:13:18.961009 19465 ts_tablet_manager.cc:1397] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260924 19:13:18.961046 19463 consensus_queue.cc:260] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:18.961977 19464 raft_consensus.cc:348] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:18.962468 19464 raft_consensus.cc:374] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:18.962615 19464 raft_consensus.cc:729] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8883cb17cd24f379f4e8298df93d088, State: Initialized, Role: FOLLOWER
I20260924 19:13:18.963404 19464 consensus_queue.cc:260] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:18.965391 19464 ts_tablet_manager.cc:1428] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 19:13:18.969509 19463 ts_tablet_manager.cc:1428] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent starting tablet: real 0.020s user 0.013s sys 0.005s
I20260924 19:13:18.971447 19465 raft_consensus.cc:348] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:18.971923 19465 raft_consensus.cc:374] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:18.972034 19465 raft_consensus.cc:729] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d4d3c6eef04c0c8223d710d363fd2e, State: Initialized, Role: FOLLOWER
I20260924 19:13:18.972729 19465 consensus_queue.cc:260] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:18.974484 19465 ts_tablet_manager.cc:1428] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260924 19:13:19.070606 19161 tablet.cc:2369] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:13:19.102721 19300 tablet.cc:2369] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:13:19.138990 19439 tablet.cc:2369] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:13:19.321550 19471 raft_consensus.cc:482] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:19.321918 19471 raft_consensus.cc:504] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:19.323616 19471 leader_election.cc:288] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207), b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411)
I20260924 19:13:19.331003 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.324183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:34957 (local address 127.11.18.66:33207)
0924 19:13:19.324416 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:19.324424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:19.325070 (+ 646us) server_negotiation.cc:407] Connection header received
0924 19:13:19.325298 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:19.325303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:19.325412 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:19.325507 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:19.326114 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.326867 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.329051 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.329979 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.330166 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.330242 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:19.330620 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 19:13:19.330749 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:13:19.331190 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.324027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:48157 (local address 127.11.18.67:46411)
0924 19:13:19.324274 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:19.324283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:19.324969 (+ 686us) server_negotiation.cc:407] Connection header received
0924 19:13:19.325209 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:19.325214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:19.325280 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:19.325423 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:19.325980 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.326719 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.329096 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.330078 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.330288 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.330370 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:19.330793 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 19:13:19.330924 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:13:19.331247 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.324148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:33207 (local address 127.11.18.65:34957)
0924 19:13:19.324954 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:19.324969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:19.325141 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:19.325524 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:19.325528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:19.325568 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:19.325955 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.325963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.327090 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.327099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.328880 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.328892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.330184 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.330190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.330413 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.330492 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 19:13:19.330597 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:13:19.330734 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":89,"thread_start_us":503,"threads_started":1}
I20260924 19:13:19.331477 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.323965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46411 (local address 127.11.18.65:48157)
0924 19:13:19.324847 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:19.324865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:19.325051 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:19.325440 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:19.325444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:19.325469 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:19.325829 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.325838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.327024 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.327034 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.328917 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.328930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.330316 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.330323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.330623 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.330650 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 19:13:19.330766 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:13:19.330908 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":167,"thread_start_us":523,"threads_started":1}
I20260924 19:13:19.332600 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d59ac6378624ff59688fbfda6a5ba8e" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8883cb17cd24f379f4e8298df93d088" is_pre_election: true
I20260924 19:13:19.332654 19249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d59ac6378624ff59688fbfda6a5ba8e" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" is_pre_election: true
I20260924 19:13:19.333240 19249 raft_consensus.cc:2454] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 0.
I20260924 19:13:19.333240 19388 raft_consensus.cc:2454] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 0.
I20260924 19:13:19.334028 19042 leader_election.cc:302] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35, b8883cb17cd24f379f4e8298df93d088, f5d4d3c6eef04c0c8223d710d363fd2e; no voters:
I20260924 19:13:19.334599 19471 raft_consensus.cc:2781] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:19.334774 19471 raft_consensus.cc:482] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:19.334877 19471 raft_consensus.cc:3037] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.337432 19471 raft_consensus.cc:504] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:19.338196 19471 leader_election.cc:288] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 election: Requested vote from peers 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207), b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411)
I20260924 19:13:19.338696 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d59ac6378624ff59688fbfda6a5ba8e" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8883cb17cd24f379f4e8298df93d088"
I20260924 19:13:19.338899 19388 raft_consensus.cc:3037] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.341602 19249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d59ac6378624ff59688fbfda6a5ba8e" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35"
I20260924 19:13:19.341791 19388 raft_consensus.cc:2454] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 1.
I20260924 19:13:19.341871 19249 raft_consensus.cc:3037] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.342403 19042 leader_election.cc:302] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8883cb17cd24f379f4e8298df93d088, f5d4d3c6eef04c0c8223d710d363fd2e; no voters:
I20260924 19:13:19.344457 19249 raft_consensus.cc:2454] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 1.
I20260924 19:13:19.344851 19471 raft_consensus.cc:2781] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:19.345430 19471 raft_consensus.cc:686] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 LEADER]: Becoming Leader. State: Replica: f5d4d3c6eef04c0c8223d710d363fd2e, State: Running, Role: LEADER
I20260924 19:13:19.345968 19471 consensus_queue.cc:237] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } }
I20260924 19:13:19.351068 18952 catalog_manager.cc:5495] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e reported cstate change: term changed from 0 to 1, leader changed from <none> to f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65). New cstate: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } health_report { overall_health: HEALTHY } } }
I20260924 19:13:19.461843 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.455686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.11.18.65:42201)
0924 19:13:19.456416 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:19.456424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:19.456442 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:13:19.456654 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:19.456657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:19.456742 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:19.456920 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:19.457362 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.458142 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.459636 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.460050 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.460185 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.460589 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:19.461138 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:19.461304 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:19.461445 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 19:13:19.461527 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":471,"threads_started":1}
I20260924 19:13:19.470126 19249 raft_consensus.cc:1261] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Refusing update from remote peer f5d4d3c6eef04c0c8223d710d363fd2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:19.470124 19388 raft_consensus.cc:1261] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Refusing update from remote peer f5d4d3c6eef04c0c8223d710d363fd2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:19.471215 19471 consensus_queue.cc:1035] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:19.471572 19474 consensus_queue.cc:1035] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:19.499876 18952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:19.500106 18952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:19.503719 18952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: Sending DeleteTablet for 3 replicas of tablet 2d59ac6378624ff59688fbfda6a5ba8e
I20260924 19:13:19.504959 19229 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d59ac6378624ff59688fbfda6a5ba8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:19 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:50096
I20260924 19:13:19.504966 19094 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d59ac6378624ff59688fbfda6a5ba8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:19 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:40908
I20260924 19:13:19.505146 19368 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d59ac6378624ff59688fbfda6a5ba8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:19 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:51640
I20260924 19:13:19.508327 18952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:13:19.510166 18952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:13:19.517896 19492 tablet_replica.cc:330] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: stopping tablet replica
I20260924 19:13:19.517896 19490 tablet_replica.cc:330] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: stopping tablet replica
I20260924 19:13:19.518507 19490 raft_consensus.cc:2229] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:19.518507 19492 raft_consensus.cc:2229] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:19.519134 19492 raft_consensus.cc:2258] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:19.519134 19490 raft_consensus.cc:2258] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:19.520612 19093 tablet_service.cc:1467] Processing CreateTablet for tablet d05d2c4be2bd42518bf9ddab610d59e4 (DEFAULT_TABLE table=test-table [id=74aecf09b4134242801953c9075b80ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:19.520850 19229 tablet_service.cc:1467] Processing CreateTablet for tablet d05d2c4be2bd42518bf9ddab610d59e4 (DEFAULT_TABLE table=test-table [id=74aecf09b4134242801953c9075b80ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:19.521221 19093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05d2c4be2bd42518bf9ddab610d59e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:19.521238 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05d2c4be2bd42518bf9ddab610d59e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:19.526008 19368 tablet_service.cc:1467] Processing CreateTablet for tablet d05d2c4be2bd42518bf9ddab610d59e4 (DEFAULT_TABLE table=test-table [id=74aecf09b4134242801953c9075b80ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:19.526522 19368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05d2c4be2bd42518bf9ddab610d59e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:19.533099 19492 ts_tablet_manager.cc:1905] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:19.536039 19490 ts_tablet_manager.cc:1905] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:19.536715 19491 tablet_replica.cc:330] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: stopping tablet replica
I20260924 19:13:19.541813 19491 raft_consensus.cc:2229] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:13:19.543423 19492 ts_tablet_manager.cc:1918] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:19.543686 19492 log.cc:1196] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/2d59ac6378624ff59688fbfda6a5ba8e
I20260924 19:13:19.543812 19491 raft_consensus.cc:2258] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:19.544420 19492 ts_tablet_manager.cc:1939] T 2d59ac6378624ff59688fbfda6a5ba8e P b8883cb17cd24f379f4e8298df93d088: Deleting consensus metadata
I20260924 19:13:19.545195 19495 tablet_bootstrap.cc:492] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: Bootstrap starting.
I20260924 19:13:19.547925 18939 catalog_manager.cc:4841] TS b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411): tablet 2d59ac6378624ff59688fbfda6a5ba8e (table test-table [id=942dee30ecc44388992b1325d6a37873]) successfully deleted
I20260924 19:13:19.548956 19495 tablet_bootstrap.cc:654] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:19.546439 19494 tablet_bootstrap.cc:492] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: Bootstrap starting.
I20260924 19:13:19.555789 19491 ts_tablet_manager.cc:1905] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:19.558615 19495 tablet_bootstrap.cc:492] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: No bootstrap required, opened a new log
I20260924 19:13:19.558843 19495 ts_tablet_manager.cc:1397] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260924 19:13:19.559715 19495 raft_consensus.cc:348] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.560053 19495 raft_consensus.cc:374] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:19.560237 19495 raft_consensus.cc:729] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8883cb17cd24f379f4e8298df93d088, State: Initialized, Role: FOLLOWER
I20260924 19:13:19.560608 19495 consensus_queue.cc:260] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.561427 19494 tablet_bootstrap.cc:654] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:19.562870 19493 tablet_bootstrap.cc:492] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: Bootstrap starting.
I20260924 19:13:19.563836 19495 ts_tablet_manager.cc:1428] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260924 19:13:19.564766 19493 tablet_bootstrap.cc:654] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:19.568120 19493 tablet_bootstrap.cc:492] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: No bootstrap required, opened a new log
I20260924 19:13:19.568333 19493 ts_tablet_manager.cc:1397] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 19:13:19.568701 19494 tablet_bootstrap.cc:492] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: No bootstrap required, opened a new log
I20260924 19:13:19.568857 19494 ts_tablet_manager.cc:1397] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20260924 19:13:19.569773 19494 raft_consensus.cc:348] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.570050 19494 raft_consensus.cc:374] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:19.570180 19494 raft_consensus.cc:729] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d4d3c6eef04c0c8223d710d363fd2e, State: Initialized, Role: FOLLOWER
I20260924 19:13:19.570458 19494 consensus_queue.cc:260] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.570911 19493 raft_consensus.cc:348] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.571187 19493 raft_consensus.cc:374] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:19.571419 19493 raft_consensus.cc:729] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762cdc846a634fffa6de3d4ffbd5ee35, State: Initialized, Role: FOLLOWER
I20260924 19:13:19.571719 19490 ts_tablet_manager.cc:1918] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:19.571887 19493 consensus_queue.cc:260] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.572590 19493 ts_tablet_manager.cc:1428] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 19:13:19.571888 19490 log.cc:1196] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/2d59ac6378624ff59688fbfda6a5ba8e
I20260924 19:13:19.573462 19490 ts_tablet_manager.cc:1939] T 2d59ac6378624ff59688fbfda6a5ba8e P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting consensus metadata
I20260924 19:13:19.575402 18940 catalog_manager.cc:4841] TS 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207): tablet 2d59ac6378624ff59688fbfda6a5ba8e (table test-table [id=942dee30ecc44388992b1325d6a37873]) successfully deleted
I20260924 19:13:19.576424 19491 ts_tablet_manager.cc:1918] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:19.576617 19491 log.cc:1196] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/2d59ac6378624ff59688fbfda6a5ba8e
I20260924 19:13:19.577090 19491 ts_tablet_manager.cc:1939] T 2d59ac6378624ff59688fbfda6a5ba8e P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting consensus metadata
I20260924 19:13:19.578055 19494 ts_tablet_manager.cc:1428] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260924 19:13:19.579706 18937 catalog_manager.cc:4841] TS f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201): tablet 2d59ac6378624ff59688fbfda6a5ba8e (table test-table [id=942dee30ecc44388992b1325d6a37873]) successfully deleted
I20260924 19:13:19.672061 19474 raft_consensus.cc:482] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:19.672340 19474 raft_consensus.cc:504] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.673094 19474 leader_election.cc:288] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207), b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411)
I20260924 19:13:19.673195 19481 raft_consensus.cc:482] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:19.673454 19481 raft_consensus.cc:504] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.673782 19249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05d2c4be2bd42518bf9ddab610d59e4" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" is_pre_election: true
I20260924 19:13:19.673979 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05d2c4be2bd42518bf9ddab610d59e4" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8883cb17cd24f379f4e8298df93d088" is_pre_election: true
I20260924 19:13:19.674057 19249 raft_consensus.cc:2454] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 0.
I20260924 19:13:19.674180 19388 raft_consensus.cc:2454] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 0.
I20260924 19:13:19.674573 19043 leader_election.cc:302] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35, f5d4d3c6eef04c0c8223d710d363fd2e; no voters:
I20260924 19:13:19.675015 19474 raft_consensus.cc:2781] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:19.675124 19474 raft_consensus.cc:482] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:19.675242 19474 raft_consensus.cc:3037] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.675431 19481 leader_election.cc:288] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201), 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207)
I20260924 19:13:19.679397 19474 raft_consensus.cc:504] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.680455 19474 leader_election.cc:288] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 election: Requested vote from peers 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207), b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411)
I20260924 19:13:19.680966 19249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05d2c4be2bd42518bf9ddab610d59e4" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35"
I20260924 19:13:19.681203 19249 raft_consensus.cc:3037] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.684508 19249 raft_consensus.cc:2454] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 1.
I20260924 19:13:19.685559 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05d2c4be2bd42518bf9ddab610d59e4" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8883cb17cd24f379f4e8298df93d088"
I20260924 19:13:19.685825 19388 raft_consensus.cc:3037] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.689116 19043 leader_election.cc:302] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35, f5d4d3c6eef04c0c8223d710d363fd2e; no voters:
I20260924 19:13:19.690074 19388 raft_consensus.cc:2454] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 1.
I20260924 19:13:19.690357 19474 raft_consensus.cc:2781] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:19.690538 19474 raft_consensus.cc:686] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 LEADER]: Becoming Leader. State: Replica: f5d4d3c6eef04c0c8223d710d363fd2e, State: Running, Role: LEADER
I20260924 19:13:19.690830 19474 consensus_queue.cc:237] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.692423 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.675914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:57169 (local address 127.11.18.65:42201)
0924 19:13:19.676160 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:19.676167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:19.681388 (+ 5221us) server_negotiation.cc:407] Connection header received
0924 19:13:19.681727 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:19.681751 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:19.681850 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:19.681929 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:19.683059 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.684609 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.688678 (+ 4069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.689998 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.690150 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.690224 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:19.691811 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0924 19:13:19.691977 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:13:19.693460 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.675635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:42201 (local address 127.11.18.67:57169)
0924 19:13:19.681234 (+ 5599us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:19.681259 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:19.681536 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:19.682195 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:19.682203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:19.682236 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:19.682806 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.682822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.685370 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.685379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.688502 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.688521 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.691412 (+ 2891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.691440 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.691618 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.691652 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:13:19.692700 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0924 19:13:19.692911 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5421,"thread_start_us":14191,"threads_started":1}
I20260924 19:13:19.694514 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05d2c4be2bd42518bf9ddab610d59e4" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" is_pre_election: true
I20260924 19:13:19.697190 18952 catalog_manager.cc:5495] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e reported cstate change: term changed from 0 to 1, leader changed from <none> to f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65). New cstate: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } health_report { overall_health: UNKNOWN } } }
I20260924 19:13:19.698165 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.676657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:58275 (local address 127.11.18.66:33207)
0924 19:13:19.676840 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:19.676846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:19.688584 (+ 11738us) server_negotiation.cc:407] Connection header received
0924 19:13:19.689401 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:19.689408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:19.689520 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:19.689639 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:19.691497 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.692751 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.696308 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.697156 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.697393 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.697467 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:19.697745 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 19:13:19.697873 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:13:19.698197 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.676084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:33207 (local address 127.11.18.67:58275)
0924 19:13:19.688468 (+ 12384us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:19.688498 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:19.688640 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:19.690788 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:19.690793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:19.690822 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:19.691291 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.691307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.693104 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.693110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.696073 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.696089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.697409 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.697415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.697586 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.697610 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:13:19.697722 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:13:19.697856 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12226,"thread_start_us":12631,"threads_started":1}
I20260924 19:13:19.699188 19249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05d2c4be2bd42518bf9ddab610d59e4" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" is_pre_election: true
I20260924 19:13:19.699513 19249 raft_consensus.cc:2379] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8883cb17cd24f379f4e8298df93d088 in current term 1: Already voted for candidate f5d4d3c6eef04c0c8223d710d363fd2e in this term.
I20260924 19:13:19.700079 19322 leader_election.cc:302] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8883cb17cd24f379f4e8298df93d088; no voters: 762cdc846a634fffa6de3d4ffbd5ee35, f5d4d3c6eef04c0c8223d710d363fd2e
I20260924 19:13:19.700708 19481 raft_consensus.cc:2726] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 24 19:13:19 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277195, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.126@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:19.766700 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.751827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36381 (local address 127.0.0.1:45550)
0924 19:13:19.752546 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:19.752573 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:19.752768 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:19.753201 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:19.753205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:19.753896 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:19.754142 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.754156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.755221 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.755235 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.756173 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.756183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.756868 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.756874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.757058 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.757842 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:19.757864 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:19.760436 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:19.764016 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:19.764031 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:19.764039 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:19.764471 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:19.764841 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:19.764845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:19.764849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:19.764984 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:19.765619 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:19.765626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:19.765783 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 19:13:19.765942 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 19:13:19.766230 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":374,"threads_started":1}
I20260924 19:13:19.766724 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.751855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.11.18.126:36381)
0924 19:13:19.752908 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:19.752912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:19.752928 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:13:19.752967 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:19.752970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:19.753066 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:19.753178 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:19.754309 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.755088 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.756326 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.756707 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.756849 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.760708 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:19.760713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:19.760717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:19.760746 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:19.763863 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:19.764592 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:19.764598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:19.764601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:19.764726 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:19.765166 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:19.765169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:19.765171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:19.765493 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:19.765641 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:19.766022 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 19:13:19.766341 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":898,"thread_start_us":727,"threads_started":1}
I20260924 19:13:19.772301 18952 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:45550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:13:19.773655 18952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:13:19.780941 19093 tablet_service.cc:1467] Processing CreateTablet for tablet 274d28f49d65420cbffddf083cf73977 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d4dfa2d32cb47f1979c0b58659460b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:13:19.781629 19093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274d28f49d65420cbffddf083cf73977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:19.789944 19229 tablet_service.cc:1467] Processing CreateTablet for tablet 274d28f49d65420cbffddf083cf73977 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d4dfa2d32cb47f1979c0b58659460b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:13:19.790220 19368 tablet_service.cc:1467] Processing CreateTablet for tablet 274d28f49d65420cbffddf083cf73977 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d4dfa2d32cb47f1979c0b58659460b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:13:19.790571 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274d28f49d65420cbffddf083cf73977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:19.790757 19368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274d28f49d65420cbffddf083cf73977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:19.795082 19494 tablet_bootstrap.cc:492] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: Bootstrap starting.
I20260924 19:13:19.797152 19494 tablet_bootstrap.cc:654] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:19.798556 19493 tablet_bootstrap.cc:492] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: Bootstrap starting.
I20260924 19:13:19.799657 19495 tablet_bootstrap.cc:492] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: Bootstrap starting.
I20260924 19:13:19.800347 19493 tablet_bootstrap.cc:654] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:19.801890 19495 tablet_bootstrap.cc:654] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:19.810382 19493 tablet_bootstrap.cc:492] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: No bootstrap required, opened a new log
I20260924 19:13:19.810603 19493 ts_tablet_manager.cc:1397] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260924 19:13:19.811425 19493 raft_consensus.cc:348] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.811883 19493 raft_consensus.cc:374] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:19.812022 19493 raft_consensus.cc:729] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762cdc846a634fffa6de3d4ffbd5ee35, State: Initialized, Role: FOLLOWER
I20260924 19:13:19.812378 19494 tablet_bootstrap.cc:492] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: No bootstrap required, opened a new log
I20260924 19:13:19.812420 19495 tablet_bootstrap.cc:492] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: No bootstrap required, opened a new log
I20260924 19:13:19.812615 19495 ts_tablet_manager.cc:1397] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260924 19:13:19.812628 19494 ts_tablet_manager.cc:1397] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20260924 19:13:19.812409 19493 consensus_queue.cc:260] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.813011 19480 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.813498 19494 raft_consensus.cc:348] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.813555 19495 raft_consensus.cc:348] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.813865 19495 raft_consensus.cc:374] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:19.813810 19494 raft_consensus.cc:374] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:19.813946 19495 raft_consensus.cc:729] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8883cb17cd24f379f4e8298df93d088, State: Initialized, Role: FOLLOWER
I20260924 19:13:19.813958 19494 raft_consensus.cc:729] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d4d3c6eef04c0c8223d710d363fd2e, State: Initialized, Role: FOLLOWER
I20260924 19:13:19.814203 19495 consensus_queue.cc:260] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.814257 19494 consensus_queue.cc:260] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.814618 19480 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:13:19.814826 19495 ts_tablet_manager.cc:1428] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:13:19.814687 19481 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.815467 19481 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:13:19.815901 19474 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.816159 19474 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:13:19.816994 19493 ts_tablet_manager.cc:1428] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260924 19:13:19.818060 19494 ts_tablet_manager.cc:1428] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:13:19.853571 19474 raft_consensus.cc:482] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:19.853819 19474 raft_consensus.cc:504] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.854669 19474 leader_election.cc:288] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207), b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411)
I20260924 19:13:19.855131 19249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274d28f49d65420cbffddf083cf73977" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" is_pre_election: true
I20260924 19:13:19.855208 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274d28f49d65420cbffddf083cf73977" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8883cb17cd24f379f4e8298df93d088" is_pre_election: true
I20260924 19:13:19.855391 19249 raft_consensus.cc:2454] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 0.
I20260924 19:13:19.855482 19388 raft_consensus.cc:2454] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 0.
I20260924 19:13:19.855872 19043 leader_election.cc:302] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35, f5d4d3c6eef04c0c8223d710d363fd2e; no voters:
I20260924 19:13:19.856171 19474 raft_consensus.cc:2781] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:19.856331 19474 raft_consensus.cc:482] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:19.856443 19474 raft_consensus.cc:3037] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.859975 19474 raft_consensus.cc:504] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.860780 19474 leader_election.cc:288] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 election: Requested vote from peers 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207), b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411)
I20260924 19:13:19.861294 19249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274d28f49d65420cbffddf083cf73977" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35"
I20260924 19:13:19.861354 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274d28f49d65420cbffddf083cf73977" candidate_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8883cb17cd24f379f4e8298df93d088"
I20260924 19:13:19.861596 19388 raft_consensus.cc:3037] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.861595 19249 raft_consensus.cc:3037] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:19.863586 19249 raft_consensus.cc:2454] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 1.
I20260924 19:13:19.864040 19043 leader_election.cc:302] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35, f5d4d3c6eef04c0c8223d710d363fd2e; no voters:
I20260924 19:13:19.864332 19474 raft_consensus.cc:2781] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:19.864495 19474 raft_consensus.cc:686] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 LEADER]: Becoming Leader. State: Replica: f5d4d3c6eef04c0c8223d710d363fd2e, State: Running, Role: LEADER
I20260924 19:13:19.864820 19388 raft_consensus.cc:2454] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d4d3c6eef04c0c8223d710d363fd2e in term 1.
I20260924 19:13:19.864820 19474 consensus_queue.cc:237] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:19.867628 19471 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: TxnStatusTablet state changed. Reason: New leader f5d4d3c6eef04c0c8223d710d363fd2e. Latest consensus state: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.867964 19471 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:13:19.868595 19513 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:13:19.868462 18952 catalog_manager.cc:5495] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e reported cstate change: term changed from 0 to 1, leader changed from <none> to f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65). New cstate: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } health_report { overall_health: UNKNOWN } } }
I20260924 19:13:19.868829 19513 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:13:19.900768 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.894971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:42201 (local address 127.0.0.1:40920)
0924 19:13:19.895221 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:19.895245 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:19.895402 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:19.895880 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:19.895885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:19.895899 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:13:19.896078 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.896087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.897271 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.897277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.898451 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.898463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.899320 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.899326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.899500 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.899562 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:13:19.900107 (+ 545us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:13:19.900131 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:13:19.900247 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:13:19.900415 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 19:13:19.900746 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.895107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.11.18.65:42201)
0924 19:13:19.895341 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:19.895348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:19.895369 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:13:19.895613 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:19.895619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:19.895726 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:19.895856 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:19.896231 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.897118 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.898608 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.899123 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.899285 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.899777 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:19.899989 (+ 212us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:19.900112 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:19.900274 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 19:13:19.900365 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 19:13:19.904850 19249 raft_consensus.cc:1261] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Refusing update from remote peer f5d4d3c6eef04c0c8223d710d363fd2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:19.904886 19388 raft_consensus.cc:1261] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Refusing update from remote peer f5d4d3c6eef04c0c8223d710d363fd2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:19.905610 19474 consensus_queue.cc:1035] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:19.906004 19471 consensus_queue.cc:1035] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:19.909447 19481 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: TxnStatusTablet state changed. Reason: New leader f5d4d3c6eef04c0c8223d710d363fd2e. Latest consensus state: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.909714 19481 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:13:19.913442 19471 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.913796 19471 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:13:19.916802 19481 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.917099 19481 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P b8883cb17cd24f379f4e8298df93d088: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:13:19.923545 19480 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: TxnStatusTablet state changed. Reason: New leader f5d4d3c6eef04c0c8223d710d363fd2e. Latest consensus state: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.923870 19480 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:13:19.926780 19482 mvcc.cc:205] Tried to move back new op lower bound from 7332975410803609600 to 7332975410649366528. Current Snapshot: MvccSnapshot[applied={T|T < 7332975410803609600}]
I20260924 19:13:19.927718 19480 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.927956 19480 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P 762cdc846a634fffa6de3d4ffbd5ee35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:13:19.928820 19471 tablet_replica.cc:439] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } } }
I20260924 19:13:19.929177 19471 tablet_replica.cc:442] T 274d28f49d65420cbffddf083cf73977 P f5d4d3c6eef04c0c8223d710d363fd2e: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:13:19.947171 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.935277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:51615 (local address 127.11.18.65:42201)
0924 19:13:19.935465 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:19.935471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:19.942034 (+ 6563us) server_negotiation.cc:407] Connection header received
0924 19:13:19.942277 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:19.942281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:19.942348 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:19.942443 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:19.942898 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.943659 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.945146 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.945796 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.945943 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.946381 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:19.946478 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:19.946616 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:19.946759 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 19:13:19.946830 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:13:19.947273 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:19.935196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:42201 (local address 127.11.18.65:51615)
0924 19:13:19.941903 (+ 6707us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:19.941926 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:19.942140 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:19.942459 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:19.942463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:19.942473 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:13:19.942752 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.942761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.943788 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.943796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.944980 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:19.944993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:19.945972 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:19.945979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:19.946119 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:19.946176 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:13:19.946630 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:13:19.946642 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:13:19.946734 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:13:19.946863 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6373,"thread_start_us":6250,"threads_started":1}
I20260924 19:13:19.960150 19388 raft_consensus.cc:1261] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Refusing update from remote peer f5d4d3c6eef04c0c8223d710d363fd2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:19.960135 19249 raft_consensus.cc:1261] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Refusing update from remote peer f5d4d3c6eef04c0c8223d710d363fd2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:19.960858 19471 consensus_queue.cc:1035] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:19.961308 19474 consensus_queue.cc:1035] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:19.970525 19479 mvcc.cc:205] Tried to move back new op lower bound from 7332975411031474176 to 7332975409936379904. Current Snapshot: MvccSnapshot[applied={T|T < 7332975411031474176}]
I20260924 19:13:20.031847 18952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:20.032094 18952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:20.035607 18952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: Sending DeleteTablet for 3 replicas of tablet d05d2c4be2bd42518bf9ddab610d59e4
I20260924 19:13:20.036660 19094 tablet_service.cc:1514] Processing DeleteTablet for tablet d05d2c4be2bd42518bf9ddab610d59e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:40908
I20260924 19:13:20.036696 19229 tablet_service.cc:1514] Processing DeleteTablet for tablet d05d2c4be2bd42518bf9ddab610d59e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:50096
I20260924 19:13:20.036971 19491 tablet_replica.cc:330] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: stopping tablet replica
I20260924 19:13:20.037088 19490 tablet_replica.cc:330] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: stopping tablet replica
I20260924 19:13:20.037084 19368 tablet_service.cc:1514] Processing DeleteTablet for tablet d05d2c4be2bd42518bf9ddab610d59e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:51640
I20260924 19:13:20.037187 19491 raft_consensus.cc:2229] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:13:20.037278 19490 raft_consensus.cc:2229] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:20.037500 19490 raft_consensus.cc:2258] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.037568 19491 raft_consensus.cc:2258] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.037835 19492 tablet_replica.cc:330] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: stopping tablet replica
I20260924 19:13:20.038048 19492 raft_consensus.cc:2229] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:20.038254 19492 raft_consensus.cc:2258] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.038909 19490 ts_tablet_manager.cc:1905] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:20.040037 19491 ts_tablet_manager.cc:1905] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:20.046308 19490 ts_tablet_manager.cc:1918] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:13:20.046496 19490 log.cc:1196] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/d05d2c4be2bd42518bf9ddab610d59e4
I20260924 19:13:20.047158 19490 ts_tablet_manager.cc:1939] T d05d2c4be2bd42518bf9ddab610d59e4 P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting consensus metadata
I20260924 19:13:20.048115 19491 ts_tablet_manager.cc:1918] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:13:20.048424 18940 catalog_manager.cc:4841] TS 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207): tablet d05d2c4be2bd42518bf9ddab610d59e4 (table test-table [id=74aecf09b4134242801953c9075b80ec]) successfully deleted
I20260924 19:13:20.048961 19491 log.cc:1196] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/d05d2c4be2bd42518bf9ddab610d59e4
I20260924 19:13:20.049189 19492 ts_tablet_manager.cc:1905] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:20.049516 19491 ts_tablet_manager.cc:1939] T d05d2c4be2bd42518bf9ddab610d59e4 P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting consensus metadata
I20260924 19:13:20.051008 18937 catalog_manager.cc:4841] TS f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201): tablet d05d2c4be2bd42518bf9ddab610d59e4 (table test-table [id=74aecf09b4134242801953c9075b80ec]) successfully deleted
I20260924 19:13:20.056358 19492 ts_tablet_manager.cc:1918] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:13:20.056530 19492 log.cc:1196] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/d05d2c4be2bd42518bf9ddab610d59e4
I20260924 19:13:20.057020 19492 ts_tablet_manager.cc:1939] T d05d2c4be2bd42518bf9ddab610d59e4 P b8883cb17cd24f379f4e8298df93d088: Deleting consensus metadata
I20260924 19:13:20.058281 18939 catalog_manager.cc:4841] TS b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411): tablet d05d2c4be2bd42518bf9ddab610d59e4 (table test-table [id=74aecf09b4134242801953c9075b80ec]) successfully deleted
Sep 24 19:13:20 dist-test-slave-0lgr krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:13:20 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:20 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:13:20.104141 19508 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:20.090489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.11.18.126:36381)
0924 19:13:20.090765 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.090772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.091123 (+ 351us) server_negotiation.cc:407] Connection header received
0924 19:13:20.091330 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.091334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.091402 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.091482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:20.093438 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.094232 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.096099 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.096564 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.096706 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.103892 (+ 7186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45556: BlockingRecv error: recv got EOF from 127.0.0.1:45556 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 24 19:13:20 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM
I20260924 19:13:20.126327 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.107577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.11.18.126:36381)
0924 19:13:20.107775 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.107782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.108225 (+ 443us) server_negotiation.cc:407] Connection header received
0924 19:13:20.108432 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.108446 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.108525 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.108631 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:20.110349 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.111468 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.113476 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.114199 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.114323 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.119373 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:20.119380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:20.119385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:20.119436 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:20.123004 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:20.123861 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:20.123866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:20.123870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:20.124058 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:20.124495 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:20.124500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:20.124504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:20.124837 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:20.125091 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.125704 (+ 613us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.125944 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:13:20.130257 18952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:13:20.131054 18952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:13:20.138285 19094 tablet_service.cc:1467] Processing CreateTablet for tablet de86ea21e3b144bb82b4fe86afd39d6b (DEFAULT_TABLE table=test-table [id=cc8774a13b294e9798ec318d2642d6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:20.138571 19368 tablet_service.cc:1467] Processing CreateTablet for tablet de86ea21e3b144bb82b4fe86afd39d6b (DEFAULT_TABLE table=test-table [id=cc8774a13b294e9798ec318d2642d6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:20.138833 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de86ea21e3b144bb82b4fe86afd39d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:20.138983 19368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de86ea21e3b144bb82b4fe86afd39d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:20.145378 19494 tablet_bootstrap.cc:492] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: Bootstrap starting.
I20260924 19:13:20.146018 19229 tablet_service.cc:1467] Processing CreateTablet for tablet de86ea21e3b144bb82b4fe86afd39d6b (DEFAULT_TABLE table=test-table [id=cc8774a13b294e9798ec318d2642d6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:20.146441 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de86ea21e3b144bb82b4fe86afd39d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:20.147097 19494 tablet_bootstrap.cc:654] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:20.149554 19494 tablet_bootstrap.cc:492] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: No bootstrap required, opened a new log
I20260924 19:13:20.149730 19494 ts_tablet_manager.cc:1397] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 19:13:20.150403 19494 raft_consensus.cc:348] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.150626 19494 raft_consensus.cc:374] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:20.150745 19494 raft_consensus.cc:729] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d4d3c6eef04c0c8223d710d363fd2e, State: Initialized, Role: FOLLOWER
I20260924 19:13:20.150985 19494 consensus_queue.cc:260] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.151641 19494 ts_tablet_manager.cc:1428] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:13:20.154831 19493 tablet_bootstrap.cc:492] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: Bootstrap starting.
I20260924 19:13:20.156136 19493 tablet_bootstrap.cc:654] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:20.156363 19495 tablet_bootstrap.cc:492] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: Bootstrap starting.
I20260924 19:13:20.157936 19495 tablet_bootstrap.cc:654] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:20.157984 19493 tablet_bootstrap.cc:492] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: No bootstrap required, opened a new log
I20260924 19:13:20.158175 19493 ts_tablet_manager.cc:1397] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 19:13:20.158720 19493 raft_consensus.cc:348] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.158998 19493 raft_consensus.cc:374] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:20.159085 19493 raft_consensus.cc:729] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762cdc846a634fffa6de3d4ffbd5ee35, State: Initialized, Role: FOLLOWER
I20260924 19:13:20.159310 19493 consensus_queue.cc:260] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.160137 19495 tablet_bootstrap.cc:492] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: No bootstrap required, opened a new log
I20260924 19:13:20.160285 19495 ts_tablet_manager.cc:1397] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260924 19:13:20.160414 19493 ts_tablet_manager.cc:1428] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 19:13:20.160959 19495 raft_consensus.cc:348] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.161173 19495 raft_consensus.cc:374] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:20.161381 19495 raft_consensus.cc:729] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8883cb17cd24f379f4e8298df93d088, State: Initialized, Role: FOLLOWER
I20260924 19:13:20.161650 19495 consensus_queue.cc:260] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.163064 19495 ts_tablet_manager.cc:1428] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:13:20.181068 19481 raft_consensus.cc:482] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:20.181294 19481 raft_consensus.cc:504] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.182138 19481 leader_election.cc:288] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201), 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207)
I20260924 19:13:20.182627 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de86ea21e3b144bb82b4fe86afd39d6b" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" is_pre_election: true
I20260924 19:13:20.182696 19248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de86ea21e3b144bb82b4fe86afd39d6b" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" is_pre_election: true
I20260924 19:13:20.182964 19248 raft_consensus.cc:2454] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8883cb17cd24f379f4e8298df93d088 in term 0.
I20260924 19:13:20.182972 19115 raft_consensus.cc:2454] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8883cb17cd24f379f4e8298df93d088 in term 0.
I20260924 19:13:20.183436 19322 leader_election.cc:302] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35, b8883cb17cd24f379f4e8298df93d088; no voters:
I20260924 19:13:20.183749 19481 raft_consensus.cc:2781] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:20.183882 19481 raft_consensus.cc:482] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:20.183971 19481 raft_consensus.cc:3037] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:20.187242 19481 raft_consensus.cc:504] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.187922 19481 leader_election.cc:288] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 election: Requested vote from peers f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201), 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207)
I20260924 19:13:20.188441 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de86ea21e3b144bb82b4fe86afd39d6b" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e"
I20260924 19:13:20.188452 19248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de86ea21e3b144bb82b4fe86afd39d6b" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35"
I20260924 19:13:20.188687 19115 raft_consensus.cc:3037] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:20.188709 19248 raft_consensus.cc:3037] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:20.190964 19248 raft_consensus.cc:2454] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8883cb17cd24f379f4e8298df93d088 in term 1.
I20260924 19:13:20.191541 19322 leader_election.cc:302] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35, b8883cb17cd24f379f4e8298df93d088; no voters:
I20260924 19:13:20.191993 19481 raft_consensus.cc:2781] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:20.192304 19115 raft_consensus.cc:2454] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8883cb17cd24f379f4e8298df93d088 in term 1.
I20260924 19:13:20.192538 19481 raft_consensus.cc:686] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 1 LEADER]: Becoming Leader. State: Replica: b8883cb17cd24f379f4e8298df93d088, State: Running, Role: LEADER
I20260924 19:13:20.193060 19481 consensus_queue.cc:237] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.196477 18952 catalog_manager.cc:5495] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8883cb17cd24f379f4e8298df93d088 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "b8883cb17cd24f379f4e8298df93d088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } health_report { overall_health: HEALTHY } } }
I20260924 19:13:20.219161 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.212166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.11.18.67:46411)
0924 19:13:20.212771 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.212776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.212793 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:20.212882 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.212886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.212961 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.213120 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:20.213619 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.214338 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.216587 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.217128 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.217235 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.217868 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:20.218526 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:20.218646 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.218792 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.218865 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":381,"threads_started":1}
I20260924 19:13:20.223309 19248 raft_consensus.cc:1261] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Refusing update from remote peer b8883cb17cd24f379f4e8298df93d088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:20.223595 19115 raft_consensus.cc:1261] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Refusing update from remote peer b8883cb17cd24f379f4e8298df93d088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:20.224013 19539 consensus_queue.cc:1035] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:20.224403 19481 consensus_queue.cc:1035] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:20.229796 19483 mvcc.cc:205] Tried to move back new op lower bound from 7332975412107341824 to 7332975411995496448. Current Snapshot: MvccSnapshot[applied={T|T < 7332975412107341824}]
I20260924 19:13:20.252699 18952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:20.252948 18952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:20.256155 18952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: Sending DeleteTablet for 3 replicas of tablet de86ea21e3b144bb82b4fe86afd39d6b
I20260924 19:13:20.257046 19094 tablet_service.cc:1514] Processing DeleteTablet for tablet de86ea21e3b144bb82b4fe86afd39d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:40908
I20260924 19:13:20.257436 19229 tablet_service.cc:1514] Processing DeleteTablet for tablet de86ea21e3b144bb82b4fe86afd39d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:50096
I20260924 19:13:20.257768 19491 tablet_replica.cc:330] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: stopping tablet replica
I20260924 19:13:20.257951 19491 raft_consensus.cc:2229] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:20.258167 19491 raft_consensus.cc:2258] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.258445 19490 tablet_replica.cc:330] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: stopping tablet replica
I20260924 19:13:20.258599 19490 raft_consensus.cc:2229] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:20.258767 19490 raft_consensus.cc:2258] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.259684 19368 tablet_service.cc:1514] Processing DeleteTablet for tablet de86ea21e3b144bb82b4fe86afd39d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:51640
I20260924 19:13:20.259976 19492 tablet_replica.cc:330] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: stopping tablet replica
I20260924 19:13:20.260157 19492 raft_consensus.cc:2229] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:13:20.260675 19492 raft_consensus.cc:2258] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.261592 18952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:13:20.262056 19491 ts_tablet_manager.cc:1905] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:13:20.262266 18952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:13:20.268628 19492 ts_tablet_manager.cc:1905] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:20.270736 19368 tablet_service.cc:1467] Processing CreateTablet for tablet ead15abc819a4be997d751bbd645e843 (DEFAULT_TABLE table=test-table [id=9e5055349ec4418097394eadb5592385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:20.271163 19368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead15abc819a4be997d751bbd645e843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:20.274034 19094 tablet_service.cc:1467] Processing CreateTablet for tablet ead15abc819a4be997d751bbd645e843 (DEFAULT_TABLE table=test-table [id=9e5055349ec4418097394eadb5592385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:20.274029 19229 tablet_service.cc:1467] Processing CreateTablet for tablet ead15abc819a4be997d751bbd645e843 (DEFAULT_TABLE table=test-table [id=9e5055349ec4418097394eadb5592385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:20.274454 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead15abc819a4be997d751bbd645e843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:20.274461 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead15abc819a4be997d751bbd645e843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:20.281177 19494 tablet_bootstrap.cc:492] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: Bootstrap starting.
I20260924 19:13:20.283008 19494 tablet_bootstrap.cc:654] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:20.283514 19490 ts_tablet_manager.cc:1905] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:20.308377 19491 ts_tablet_manager.cc:1918] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:20.313676 19495 tablet_bootstrap.cc:492] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: Bootstrap starting.
I20260924 19:13:20.316572 19495 tablet_bootstrap.cc:654] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:20.322068 19495 tablet_bootstrap.cc:492] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: No bootstrap required, opened a new log
I20260924 19:13:20.322414 19495 ts_tablet_manager.cc:1397] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:13:20.323554 19495 raft_consensus.cc:348] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.324798 19495 raft_consensus.cc:374] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:20.325001 19495 raft_consensus.cc:729] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8883cb17cd24f379f4e8298df93d088, State: Initialized, Role: FOLLOWER
I20260924 19:13:20.325543 19495 consensus_queue.cc:260] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.329861 19495 ts_tablet_manager.cc:1428] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260924 19:13:20.334632 19493 tablet_bootstrap.cc:492] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: Bootstrap starting.
I20260924 19:13:20.337249 19493 tablet_bootstrap.cc:654] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:20.344208 19490 ts_tablet_manager.cc:1918] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:20.346128 19492 ts_tablet_manager.cc:1918] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:20.346508 19491 log.cc:1196] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/de86ea21e3b144bb82b4fe86afd39d6b
I20260924 19:13:20.349705 19492 log.cc:1196] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/de86ea21e3b144bb82b4fe86afd39d6b
I20260924 19:13:20.347407 19491 ts_tablet_manager.cc:1939] T de86ea21e3b144bb82b4fe86afd39d6b P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting consensus metadata
I20260924 19:13:20.351775 18937 catalog_manager.cc:4841] TS f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201): tablet de86ea21e3b144bb82b4fe86afd39d6b (table test-table [id=cc8774a13b294e9798ec318d2642d6ea]) successfully deleted
I20260924 19:13:20.352849 19492 ts_tablet_manager.cc:1939] T de86ea21e3b144bb82b4fe86afd39d6b P b8883cb17cd24f379f4e8298df93d088: Deleting consensus metadata
I20260924 19:13:20.354003 19490 log.cc:1196] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/de86ea21e3b144bb82b4fe86afd39d6b
I20260924 19:13:20.354892 19490 ts_tablet_manager.cc:1939] T de86ea21e3b144bb82b4fe86afd39d6b P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting consensus metadata
I20260924 19:13:20.357429 18939 catalog_manager.cc:4841] TS b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411): tablet de86ea21e3b144bb82b4fe86afd39d6b (table test-table [id=cc8774a13b294e9798ec318d2642d6ea]) successfully deleted
I20260924 19:13:20.360386 19494 tablet_bootstrap.cc:492] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: No bootstrap required, opened a new log
I20260924 19:13:20.360594 19494 ts_tablet_manager.cc:1397] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent bootstrapping tablet: real 0.079s user 0.024s sys 0.050s
I20260924 19:13:20.361680 19493 tablet_bootstrap.cc:492] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: No bootstrap required, opened a new log
I20260924 19:13:20.361541 19494 raft_consensus.cc:348] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.361876 19494 raft_consensus.cc:374] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:20.361974 19494 raft_consensus.cc:729] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d4d3c6eef04c0c8223d710d363fd2e, State: Initialized, Role: FOLLOWER
I20260924 19:13:20.362025 19493 ts_tablet_manager.cc:1397] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.006s
I20260924 19:13:20.362293 19494 consensus_queue.cc:260] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.362982 19493 raft_consensus.cc:348] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.363336 19493 raft_consensus.cc:374] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:20.363484 19493 raft_consensus.cc:729] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762cdc846a634fffa6de3d4ffbd5ee35, State: Initialized, Role: FOLLOWER
I20260924 19:13:20.363821 19493 consensus_queue.cc:260] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.365670 19494 ts_tablet_manager.cc:1428] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 19:13:20.366858 19493 ts_tablet_manager.cc:1428] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:13:20.370294 18940 catalog_manager.cc:4841] TS 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207): tablet de86ea21e3b144bb82b4fe86afd39d6b (table test-table [id=cc8774a13b294e9798ec318d2642d6ea]) successfully deleted
I20260924 19:13:20.384181 19480 raft_consensus.cc:482] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:20.384430 19480 raft_consensus.cc:504] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.386914 19480 leader_election.cc:288] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201), b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411)
I20260924 19:13:20.401297 19481 raft_consensus.cc:482] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:20.401546 19481 raft_consensus.cc:504] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.402403 19481 leader_election.cc:288] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201), 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207)
I20260924 19:13:20.403090 19248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead15abc819a4be997d751bbd645e843" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" is_pre_election: true
I20260924 19:13:20.403350 19248 raft_consensus.cc:2454] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8883cb17cd24f379f4e8298df93d088 in term 0.
I20260924 19:13:20.404188 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead15abc819a4be997d751bbd645e843" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" is_pre_election: true
I20260924 19:13:20.404295 19322 leader_election.cc:302] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35, b8883cb17cd24f379f4e8298df93d088; no voters:
I20260924 19:13:20.404453 19115 raft_consensus.cc:2454] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8883cb17cd24f379f4e8298df93d088 in term 0.
I20260924 19:13:20.404630 19481 raft_consensus.cc:2781] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:20.404758 19481 raft_consensus.cc:482] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:20.404896 19481 raft_consensus.cc:3037] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:20.408438 19481 raft_consensus.cc:504] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.409655 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead15abc819a4be997d751bbd645e843" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e"
I20260924 19:13:20.409906 19115 raft_consensus.cc:3037] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:20.410081 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.387855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:53031 (local address 127.11.18.67:46411)
0924 19:13:20.388087 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.388096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.400936 (+ 12840us) server_negotiation.cc:407] Connection header received
0924 19:13:20.401488 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.401494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.401586 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.401653 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:20.402688 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.403906 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.407145 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.408707 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.408861 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.408939 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.409608 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.409774 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:13:20.411075 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.387414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46411 (local address 127.11.18.66:53031)
0924 19:13:20.400824 (+ 13410us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:20.400845 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:20.401288 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:20.402049 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:20.402054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:20.402080 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:20.402446 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:20.402459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.404208 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.404213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:20.406924 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:20.406939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.409172 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.409178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:20.409409 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.409444 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:13:20.410495 (+ 1051us) client_negotiation.cc:240] Negotiation successful
0924 19:13:20.410650 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13268,"thread_start_us":13040,"threads_started":1}
I20260924 19:13:20.413041 19115 raft_consensus.cc:2454] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8883cb17cd24f379f4e8298df93d088 in term 1.
I20260924 19:13:20.413637 19319 leader_election.cc:302] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8883cb17cd24f379f4e8298df93d088, f5d4d3c6eef04c0c8223d710d363fd2e; no voters:
I20260924 19:13:20.413944 19539 raft_consensus.cc:2781] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:20.414016 19248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead15abc819a4be997d751bbd645e843" candidate_uuid: "b8883cb17cd24f379f4e8298df93d088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762cdc846a634fffa6de3d4ffbd5ee35"
I20260924 19:13:20.414394 19248 raft_consensus.cc:3037] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:20.417189 19248 raft_consensus.cc:2454] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8883cb17cd24f379f4e8298df93d088 in term 1.
I20260924 19:13:20.417685 19481 leader_election.cc:288] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [CANDIDATE]: Term 1 election: Requested vote from peers f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201), 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207)
I20260924 19:13:20.418579 19388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead15abc819a4be997d751bbd645e843" candidate_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8883cb17cd24f379f4e8298df93d088" is_pre_election: true
I20260924 19:13:20.422462 19539 raft_consensus.cc:686] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 1 LEADER]: Becoming Leader. State: Replica: b8883cb17cd24f379f4e8298df93d088, State: Running, Role: LEADER
I20260924 19:13:20.422801 19539 consensus_queue.cc:237] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } }
I20260924 19:13:20.424779 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.387426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:35191 (local address 127.11.18.65:42201)
0924 19:13:20.387651 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.387659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.399080 (+ 11421us) server_negotiation.cc:407] Connection header received
0924 19:13:20.399188 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.399198 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.399310 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.399431 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:20.419143 (+ 19712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.420140 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.422383 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.423708 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.423878 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.423965 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.424366 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.424491 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:13:20.424938 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.387087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:42201 (local address 127.11.18.66:35191)
0924 19:13:20.398863 (+ 11776us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:20.398886 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:20.399037 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:20.400691 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:20.400696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:20.400726 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:20.401073 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:20.418919 (+ 17846us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.420401 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.420405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:20.422203 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:20.422216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.424008 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.424017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:20.424191 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.424216 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:13:20.424317 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:13:20.424457 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11634,"thread_start_us":13006,"threads_started":1}
I20260924 19:13:20.425982 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead15abc819a4be997d751bbd645e843" candidate_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" is_pre_election: true
I20260924 19:13:20.426360 19115 raft_consensus.cc:2379] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 762cdc846a634fffa6de3d4ffbd5ee35 in current term 1: Already voted for candidate b8883cb17cd24f379f4e8298df93d088 in this term.
I20260924 19:13:20.431756 18953 catalog_manager.cc:5495] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8883cb17cd24f379f4e8298df93d088 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "b8883cb17cd24f379f4e8298df93d088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8883cb17cd24f379f4e8298df93d088" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46411 } health_report { overall_health: HEALTHY } } }
I20260924 19:13:20.434834 19182 leader_election.cc:302] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 762cdc846a634fffa6de3d4ffbd5ee35; no voters: b8883cb17cd24f379f4e8298df93d088, f5d4d3c6eef04c0c8223d710d363fd2e
I20260924 19:13:20.435298 19480 raft_consensus.cc:2726] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:13:20.473630 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.467942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:46981 (local address 127.11.18.65:42201)
0924 19:13:20.468140 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.468147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.468409 (+ 262us) server_negotiation.cc:407] Connection header received
0924 19:13:20.468674 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.468679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.468751 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.468884 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:20.469433 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.470201 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.471737 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.472170 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.472411 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.472847 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:20.472961 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:20.473066 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.473203 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.473290 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:13:20.473814 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.467589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:42201 (local address 127.11.18.67:46981)
0924 19:13:20.468309 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:20.468329 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:20.468518 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:20.468888 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:20.468892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:20.468904 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:13:20.469235 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:20.469245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.470365 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.470375 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:20.471591 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:20.471603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.472449 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.472456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:20.472590 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.472645 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:13:20.473065 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:13:20.473078 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:13:20.473183 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:13:20.473325 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":565,"threads_started":1}
I20260924 19:13:20.485456 19248 raft_consensus.cc:1261] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Refusing update from remote peer b8883cb17cd24f379f4e8298df93d088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:20.486224 19539 consensus_queue.cc:1035] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "762cdc846a634fffa6de3d4ffbd5ee35" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:20.486604 19115 raft_consensus.cc:1261] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Refusing update from remote peer b8883cb17cd24f379f4e8298df93d088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:20.487556 19539 consensus_queue.cc:1035] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d4d3c6eef04c0c8223d710d363fd2e" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:20.535020 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.526224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46411 (local address 127.11.18.65:55621)
0924 19:13:20.527145 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:20.527173 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:20.527360 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:20.528214 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:20.528217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:20.528228 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:13:20.528490 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:20.528499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.529861 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.529867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:20.532447 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:20.532466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.533545 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.533554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:20.533732 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.533772 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:13:20.534276 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:13:20.534291 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:13:20.534397 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:13:20.534549 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":588,"threads_started":1}
I20260924 19:13:20.535790 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.526479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:55621 (local address 127.11.18.67:46411)
0924 19:13:20.526665 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.526671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.527252 (+ 581us) server_negotiation.cc:407] Connection header received
0924 19:13:20.527532 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.527538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.527630 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.527699 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:20.528647 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.529718 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.532625 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.533260 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.533506 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.533976 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:20.534148 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:20.534270 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.534420 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.534500 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:13:20.608477 18952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:20.608789 18952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:20.612416 18952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc8f661014fe455bb862a5ced5156261: Sending DeleteTablet for 3 replicas of tablet ead15abc819a4be997d751bbd645e843
I20260924 19:13:20.613606 19229 tablet_service.cc:1514] Processing DeleteTablet for tablet ead15abc819a4be997d751bbd645e843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:50096
I20260924 19:13:20.613945 19490 tablet_replica.cc:330] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: stopping tablet replica
I20260924 19:13:20.614166 19490 raft_consensus.cc:2229] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:20.614372 19490 raft_consensus.cc:2258] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.614447 19368 tablet_service.cc:1514] Processing DeleteTablet for tablet ead15abc819a4be997d751bbd645e843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:51640
I20260924 19:13:20.614732 19492 tablet_replica.cc:330] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: stopping tablet replica
I20260924 19:13:20.614908 19492 raft_consensus.cc:2229] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:13:20.615265 19492 raft_consensus.cc:2258] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.616570 19490 ts_tablet_manager.cc:1905] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:20.622694 19492 ts_tablet_manager.cc:1905] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:20.623278 19094 tablet_service.cc:1514] Processing DeleteTablet for tablet ead15abc819a4be997d751bbd645e843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:20 UTC) from {username='oryx', principal='oryx/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:40908
I20260924 19:13:20.623637 19491 tablet_replica.cc:330] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: stopping tablet replica
I20260924 19:13:20.623872 19491 raft_consensus.cc:2229] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:20.624102 19491 raft_consensus.cc:2258] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:20.625691 19490 ts_tablet_manager.cc:1918] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:13:20.625883 19490 log.cc:1196] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/ead15abc819a4be997d751bbd645e843
I20260924 19:13:20.626345 19490 ts_tablet_manager.cc:1939] T ead15abc819a4be997d751bbd645e843 P 762cdc846a634fffa6de3d4ffbd5ee35: Deleting consensus metadata
I20260924 19:13:20.627480 18940 catalog_manager.cc:4841] TS 762cdc846a634fffa6de3d4ffbd5ee35 (127.11.18.66:33207): tablet ead15abc819a4be997d751bbd645e843 (table test-table [id=9e5055349ec4418097394eadb5592385]) successfully deleted
I20260924 19:13:20.628569 19491 ts_tablet_manager.cc:1905] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:20.630568 19492 ts_tablet_manager.cc:1918] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:13:20.630777 19492 log.cc:1196] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/ead15abc819a4be997d751bbd645e843
I20260924 19:13:20.631269 19492 ts_tablet_manager.cc:1939] T ead15abc819a4be997d751bbd645e843 P b8883cb17cd24f379f4e8298df93d088: Deleting consensus metadata
I20260924 19:13:20.632556 18939 catalog_manager.cc:4841] TS b8883cb17cd24f379f4e8298df93d088 (127.11.18.67:46411): tablet ead15abc819a4be997d751bbd645e843 (table test-table [id=9e5055349ec4418097394eadb5592385]) successfully deleted
I20260924 19:13:20.633637 19491 ts_tablet_manager.cc:1918] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:13:20.633795 19491 log.cc:1196] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/ead15abc819a4be997d751bbd645e843
I20260924 19:13:20.634212 19491 ts_tablet_manager.cc:1939] T ead15abc819a4be997d751bbd645e843 P f5d4d3c6eef04c0c8223d710d363fd2e: Deleting consensus metadata
I20260924 19:13:20.635186 18937 catalog_manager.cc:4841] TS f5d4d3c6eef04c0c8223d710d363fd2e (127.11.18.65:42201): tablet ead15abc819a4be997d751bbd645e843 (table test-table [id=9e5055349ec4418097394eadb5592385]) successfully deleted
I20260924 19:13:20.636420 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 19024
Sep 24 19:13:20 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.65@KRBTEST.COM
I20260924 19:13:20.652364 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.637566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.11.18.65:42201)
0924 19:13:20.637749 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.637755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.637886 (+ 131us) server_negotiation.cc:407] Connection header received
0924 19:13:20.638115 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.638121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.638192 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.638299 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:20.639453 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.640186 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.641497 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.641929 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.642065 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.646200 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:20.646216 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:20.646230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:20.646272 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:20.649123 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:20.649902 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:20.649908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:20.649911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:20.650045 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:20.650625 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:20.650632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:20.650637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:20.651037 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:20.651512 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.651690 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.651941 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:13:20.653110 19155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:20.807930 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 19024
I20260924 19:13:20.835273 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 19164
Sep 24 19:13:20 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.66@KRBTEST.COM
I20260924 19:13:20.851889 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.836189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.11.18.66:33207)
0924 19:13:20.836778 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.836784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.836804 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:13:20.836872 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.836880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.836965 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.837092 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:20.838430 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.839180 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.840518 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.840956 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.841094 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.845763 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:20.845778 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:20.845796 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:20.845831 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:20.848735 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:20.849549 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:20.849556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:20.849558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:20.849719 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:20.850189 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:20.850195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:20.850198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:20.850570 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:20.850844 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.851431 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.851604 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":587,"threads_started":1}
I20260924 19:13:20.852531 19294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:20.959863 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 19164
I20260924 19:13:20.978006 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 19303
Sep 24 19:13:20 dist-test-slave-0lgr krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.18.67@KRBTEST.COM
I20260924 19:13:20.993714 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:20.978784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.11.18.67:46411)
0924 19:13:20.978978 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:20.978986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:20.979077 (+ 91us) server_negotiation.cc:407] Connection header received
0924 19:13:20.979310 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:20.979322 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:20.979385 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:20.979493 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:20.980781 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.981576 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.982820 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:20.983234 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:20.983363 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:20.987908 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:20.987924 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:20.987941 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:20.987992 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:20.990938 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:20.991599 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:20.991606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:20.991608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:20.991756 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:20.992269 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:20.992273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:20.992275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:20.992652 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:20.992900 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:20.993174 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 19:13:20.993476 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:13:20.994395 19433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:21.118499 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 19303
I20260924 19:13:21.141671 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 18921
I20260924 19:13:21.143002 18993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:21.261660 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 18921
2026-09-24T19:13:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7192 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:13:21 dist-test-slave-0lgr krb5kdc[19569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:13:21 dist-test-slave-0lgr krb5kdc[19569](info): set up 2 sockets
Sep 24 19:13:21 dist-test-slave-0lgr krb5kdc[19569](info): commencing operation
krb5kdc: starting...
W20260924 19:13:22.414424 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:13:22 dist-test-slave-0lgr krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:13:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:13:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:13:22.658231 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41705
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:41705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:22.989043 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:22.989404 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:22.989500 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:22.996816 19582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:22.996959 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:22.997008 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:22.997059 19582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:22.997104 19582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:23.007025 19582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:41705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:23.008270 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:23.009634 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:23.022396 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:23.022612 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:23.022639 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:23.027073 19582 server_base.cc:1034] running on GCE node
I20260924 19:13:23.028095 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:23.029883 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:23.031270 19582 hybrid_clock.cc:648] HybridClock initialized: now 1790277203031203 us; error 58 us; skew 500 ppm
I20260924 19:13:23.031625 19582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T19:13:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1844 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:13:23 dist-test-slave-0lgr krb5kdc[19596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:13:23 dist-test-slave-0lgr krb5kdc[19596](info): set up 2 sockets
Sep 24 19:13:23 dist-test-slave-0lgr krb5kdc[19596](info): commencing operation
krb5kdc: starting...
W20260924 19:13:24.242965 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:13:24 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:13:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:13:24.485842 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45179
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:24.820487 19609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:24.820820 19609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:24.820996 19609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:24.828401 19609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:24.828517 19609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:24.828593 19609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:24.828701 19609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:24.828768 19609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:24.838831 19609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:24.840193 19609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:24.841552 19609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:24.853678 19615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:24.853705 19617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:24.853739 19614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:24.854872 19609 server_base.cc:1034] running on GCE node
I20260924 19:13:24.855712 19609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:24.857431 19609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:24.858682 19609 hybrid_clock.cc:648] HybridClock initialized: now 1790277204858620 us; error 56 us; skew 500 ppm
I20260924 19:13:24.859040 19609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:24 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277204, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:24.868989 19609 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:13:24.870796 19609 webserver.cc:460] Webserver started at http://127.11.18.126:46305/ using document root <none> and password file <none>
I20260924 19:13:24.871520 19609 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:24.871672 19609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:24.871981 19609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:24.875082 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "f21ba721014940a5992be9eabcfa5691"
format_stamp: "Formatted at 2026-09-24 19:13:24 on dist-test-slave-0lgr"
server_key: "82176ba25133c4214d9a8372dbc16a0f"
server_key_iv: "b6b90414849b7d41f4e68d39294b4bc2"
server_key_version: "encryptionkey@0"
I20260924 19:13:24.875896 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "f21ba721014940a5992be9eabcfa5691"
format_stamp: "Formatted at 2026-09-24 19:13:24 on dist-test-slave-0lgr"
server_key: "82176ba25133c4214d9a8372dbc16a0f"
server_key_iv: "b6b90414849b7d41f4e68d39294b4bc2"
server_key_version: "encryptionkey@0"
I20260924 19:13:24.881568 19609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:13:24.885215 19624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:24.886754 19609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:13:24.886957 19609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "f21ba721014940a5992be9eabcfa5691"
format_stamp: "Formatted at 2026-09-24 19:13:24 on dist-test-slave-0lgr"
server_key: "82176ba25133c4214d9a8372dbc16a0f"
server_key_iv: "b6b90414849b7d41f4e68d39294b4bc2"
server_key_version: "encryptionkey@0"
I20260924 19:13:24.887233 19609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:24.908674 19609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:24.913259 19609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:24.913625 19609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:24.933488 19609 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:45179
I20260924 19:13:24.933533 19675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:45179 every 8 connection(s)
I20260924 19:13:24.934871 19609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:13:24.938903 19676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:24.939188 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 19609
I20260924 19:13:24.939424 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:13:24.940099 11337 external_mini_cluster.cc:1428] Setting key a83d41887b19ee0b67b0a958f1eb4025
I20260924 19:13:24.954628 19676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Bootstrap starting.
Sep 24 19:13:24 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:24.958077 19676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:24.959511 19676 log.cc:824] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:24.963025 19676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: No bootstrap required, opened a new log
I20260924 19:13:24.966787 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:24.942408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.11.18.126:45179)
0924 19:13:24.943266 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:24.943282 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:24.943334 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:13:24.946249 (+ 2915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:24.946281 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:24.946874 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:24.947376 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:24.948747 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:24.950833 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:24.952217 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:24.953280 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:24.953486 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:24.957997 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:24.958009 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:24.958031 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:24.958100 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:24.961916 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:24.962660 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:24.962670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:24.962677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:24.962880 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:24.963408 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:24.963412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:24.963415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:24.964142 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:24.964435 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:24.964843 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 19:13:24.965366 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":357,"threads_started":1}
I20260924 19:13:24.971871 19676 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } }
I20260924 19:13:24.972254 19676 raft_consensus.cc:374] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:24.972355 19676 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21ba721014940a5992be9eabcfa5691, State: Initialized, Role: FOLLOWER
I20260924 19:13:24.972894 19676 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } }
I20260924 19:13:24.973133 19676 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:13:24.973261 19676 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:13:24.973460 19676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:24.975646 19676 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } }
I20260924 19:13:24.976171 19676 leader_election.cc:302] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21ba721014940a5992be9eabcfa5691; no voters:
I20260924 19:13:24.976814 19676 leader_election.cc:288] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:13:24.976980 19681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:24.977702 19681 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 1 LEADER]: Becoming Leader. State: Replica: f21ba721014940a5992be9eabcfa5691, State: Running, Role: LEADER
I20260924 19:13:24.978216 19681 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } }
I20260924 19:13:24.979074 19676 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:24.981277 19682 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f21ba721014940a5992be9eabcfa5691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } } }
I20260924 19:13:24.981230 19683 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21ba721014940a5992be9eabcfa5691. Latest consensus state: current_term: 1 leader_uuid: "f21ba721014940a5992be9eabcfa5691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } } }
I20260924 19:13:24.981742 19683 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:24.981740 19682 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:24.983875 19686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:13:24.989156 19686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:13:24.998682 19686 catalog_manager.cc:1347] Generated new cluster ID: 25d1775e02c942e7b12de81f69827a61
I20260924 19:13:24.998811 19686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:13:25.009539 19686 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:13:25.010938 19686 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:13:25.028369 19686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Generated new TSK 0
I20260924 19:13:25.029296 19686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:13:25.094238 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45179
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:25.428714 19702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:25.429016 19702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:25.429111 19702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:25.436545 19702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:25.436671 19702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:25.436786 19702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:25.446549 19702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:25.447806 19702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:25.449040 19702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:25.462836 19707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:25.462891 19708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:25.463400 19710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:25.463948 19702 server_base.cc:1034] running on GCE node
I20260924 19:13:25.464660 19702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:25.465771 19702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:25.467024 19702 hybrid_clock.cc:648] HybridClock initialized: now 1790277205466947 us; error 74 us; skew 500 ppm
I20260924 19:13:25.467394 19702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:25 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277205, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:25.471329 19702 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:13:25.473001 19702 webserver.cc:460] Webserver started at http://127.11.18.65:32811/ using document root <none> and password file <none>
I20260924 19:13:25.473827 19702 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:25.473940 19702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:25.474200 19702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:25.477283 19702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "5fbbc292916341999b4f88847f11343d"
format_stamp: "Formatted at 2026-09-24 19:13:25 on dist-test-slave-0lgr"
server_key: "8741c0b69f7744ed0905b25d476c5459"
server_key_iv: "c0f43f1b50fe2b44316057012737cd8d"
server_key_version: "encryptionkey@0"
I20260924 19:13:25.478211 19702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "5fbbc292916341999b4f88847f11343d"
format_stamp: "Formatted at 2026-09-24 19:13:25 on dist-test-slave-0lgr"
server_key: "8741c0b69f7744ed0905b25d476c5459"
server_key_iv: "c0f43f1b50fe2b44316057012737cd8d"
server_key_version: "encryptionkey@0"
I20260924 19:13:25.483994 19702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:13:25.488605 19717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:25.490796 19702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:13:25.491130 19702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "5fbbc292916341999b4f88847f11343d"
format_stamp: "Formatted at 2026-09-24 19:13:25 on dist-test-slave-0lgr"
server_key: "8741c0b69f7744ed0905b25d476c5459"
server_key_iv: "c0f43f1b50fe2b44316057012737cd8d"
server_key_version: "encryptionkey@0"
I20260924 19:13:25.491397 19702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:25.513271 19702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:25.518328 19702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:25.518699 19702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:25.519961 19702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:25.521497 19702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:25.521579 19702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:25.521718 19702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:25.521786 19702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:25.558877 19702 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:40955
I20260924 19:13:25.558919 19829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:40955 every 8 connection(s)
I20260924 19:13:25.560384 19702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:25.566584 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 19702
I20260924 19:13:25.566780 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:13:25.567592 11337 external_mini_cluster.cc:1428] Setting key ad6bea9cb55d6ec7232f98776d467e73
Sep 24 19:13:25 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277205, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:25.584219 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:25.563299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:36649 (local address 127.11.18.126:45179)
0924 19:13:25.563807 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:25.563812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:25.564869 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 19:13:25.567030 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:25.567036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:25.567147 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:25.567274 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:25.569298 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:25.570487 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:25.572192 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:25.572884 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:25.573031 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:25.576823 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:25.576832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:25.576837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:25.576883 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:25.580609 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:25.581514 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:25.581522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:25.581528 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:25.581683 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:25.582114 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:25.582121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:25.582126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:25.582512 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:25.582682 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:25.583418 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 19:13:25.583683 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260924 19:13:25.585222 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:25.563902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:36649)
0924 19:13:25.564678 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:25.564721 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:25.566638 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:25.567550 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:25.567559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:25.568263 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:25.569090 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:25.569117 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:25.570677 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:25.570681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:25.572034 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:25.572046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:25.573212 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:25.573218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:25.573599 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:25.574311 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:25.574339 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:25.576538 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:25.580782 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:25.580790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:25.580805 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:25.581381 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:25.581796 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:25.581800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:25.581802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:25.581993 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:25.582655 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:25.582663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:25.583010 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 19:13:25.583262 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 19:13:25.583738 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
I20260924 19:13:25.587069 19830 heartbeater.cc:347] Connected to a master server at 127.11.18.126:45179
I20260924 19:13:25.587423 19830 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:25.588436 19830 heartbeater.cc:511] Master 127.11.18.126:45179 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:13:25.591161 19641 ts_manager.cc:194] Registered new tserver with Master: 5fbbc292916341999b4f88847f11343d (127.11.18.65:40955)
I20260924 19:13:25.593032 19641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:36649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:13:25.622809 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45179
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:25.993433 19836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:25.993973 19836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:25.994161 19836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:26.002449 19836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:26.002573 19836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:26.002687 19836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:26.013288 19836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:26.014784 19836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:26.016084 19836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:26.030483 19841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:26.030977 19844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:26.031286 19842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:26.035321 19836 server_base.cc:1034] running on GCE node
I20260924 19:13:26.035965 19836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:26.037047 19836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:26.038331 19836 hybrid_clock.cc:648] HybridClock initialized: now 1790277206038240 us; error 84 us; skew 500 ppm
I20260924 19:13:26.038729 19836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:26 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277206, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:26.042407 19836 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:13:26.044080 19836 webserver.cc:460] Webserver started at http://127.11.18.66:37923/ using document root <none> and password file <none>
I20260924 19:13:26.044826 19836 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:26.044973 19836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:26.045317 19836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:26.048709 19836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "cf7720bb3bdb4e51b6619f961157f87e"
format_stamp: "Formatted at 2026-09-24 19:13:26 on dist-test-slave-0lgr"
server_key: "f93c30aeb1ba7309083b89e4f1d93ecd"
server_key_iv: "045eed3ff9343848ae592724ce597d90"
server_key_version: "encryptionkey@0"
I20260924 19:13:26.049629 19836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "cf7720bb3bdb4e51b6619f961157f87e"
format_stamp: "Formatted at 2026-09-24 19:13:26 on dist-test-slave-0lgr"
server_key: "f93c30aeb1ba7309083b89e4f1d93ecd"
server_key_iv: "045eed3ff9343848ae592724ce597d90"
server_key_version: "encryptionkey@0"
I20260924 19:13:26.055559 19836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:13:26.059556 19851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:26.060971 19836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:13:26.061195 19836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "cf7720bb3bdb4e51b6619f961157f87e"
format_stamp: "Formatted at 2026-09-24 19:13:26 on dist-test-slave-0lgr"
server_key: "f93c30aeb1ba7309083b89e4f1d93ecd"
server_key_iv: "045eed3ff9343848ae592724ce597d90"
server_key_version: "encryptionkey@0"
I20260924 19:13:26.061468 19836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:26.073190 19836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:26.077773 19836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:26.078101 19836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:26.079252 19836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:26.080797 19836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:26.080899 19836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:26.081055 19836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:26.081130 19836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:26.117046 19836 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:46353
I20260924 19:13:26.117086 19963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:46353 every 8 connection(s)
I20260924 19:13:26.118422 19836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:26.126902 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 19836
I20260924 19:13:26.127094 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:13:26.127517 11337 external_mini_cluster.cc:1428] Setting key d3161a849b9059232211a3cedbf314e7
Sep 24 19:13:26 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277206, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:26.143201 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:26.121447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:44659 (local address 127.11.18.126:45179)
0924 19:13:26.122017 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:26.122023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:26.123192 (+ 1169us) server_negotiation.cc:407] Connection header received
0924 19:13:26.125524 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:26.125532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:26.125650 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:26.125786 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:26.127740 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.128580 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.130283 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.130712 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.130853 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:26.134765 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:26.134774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:26.134782 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:26.134829 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:26.138418 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:26.139684 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:26.139693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:26.139699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:26.139873 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:26.140805 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:26.140813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:26.140820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:26.141238 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:26.141446 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:26.142429 (+ 983us) server_negotiation.cc:299] Negotiation successful
0924 19:13:26.142717 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":281,"threads_started":1}
I20260924 19:13:26.143929 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:26.122045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:44659)
0924 19:13:26.122985 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:26.123033 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:26.125119 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:26.126243 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:26.126252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:26.126817 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:26.127509 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:26.127530 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.128750 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.128755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:26.130125 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:26.130143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.130948 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.130955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:26.131190 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:26.131903 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:26.131926 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:26.134508 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:26.138621 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:26.138629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:26.138644 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:26.139479 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:26.140174 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:26.140181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:26.140186 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:26.140598 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:26.141497 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:26.141504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:26.141913 (+ 409us) client_negotiation.cc:769] Sending connection context
0924 19:13:26.142270 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 19:13:26.142707 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":333,"threads_started":1}
I20260924 19:13:26.146144 19964 heartbeater.cc:347] Connected to a master server at 127.11.18.126:45179
I20260924 19:13:26.146638 19964 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:26.147701 19964 heartbeater.cc:511] Master 127.11.18.126:45179 requested a full tablet report, sending...
I20260924 19:13:26.149899 19641 ts_manager.cc:194] Registered new tserver with Master: cf7720bb3bdb4e51b6619f961157f87e (127.11.18.66:46353)
I20260924 19:13:26.150852 19641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:44659
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:13:26.186601 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45179
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:26.523334 19970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:26.523655 19970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:26.523751 19970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:26.531201 19970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:26.531327 19970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:26.531440 19970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:26.542572 19970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:26.543951 19970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:26.545343 19970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:26.559979 19975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:26.560590 19978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:26.560781 19976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:26.562269 19970 server_base.cc:1034] running on GCE node
I20260924 19:13:26.563023 19970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:26.564210 19970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:26.565438 19970 hybrid_clock.cc:648] HybridClock initialized: now 1790277206565364 us; error 58 us; skew 500 ppm
I20260924 19:13:26.565794 19970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:26 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277206, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:26.569310 19970 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:13:26.571219 19970 webserver.cc:460] Webserver started at http://127.11.18.67:42369/ using document root <none> and password file <none>
I20260924 19:13:26.571993 19970 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:26.572160 19970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:26.572500 19970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:26.575743 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "fe97477e6cb04b2ebfa2263e1fe9c9fd"
format_stamp: "Formatted at 2026-09-24 19:13:26 on dist-test-slave-0lgr"
server_key: "37fe6cd1492b41f3a60096ccb9265e48"
server_key_iv: "3efa40a6b065d719487775965fb1be2a"
server_key_version: "encryptionkey@0"
I20260924 19:13:26.576622 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "fe97477e6cb04b2ebfa2263e1fe9c9fd"
format_stamp: "Formatted at 2026-09-24 19:13:26 on dist-test-slave-0lgr"
server_key: "37fe6cd1492b41f3a60096ccb9265e48"
server_key_iv: "3efa40a6b065d719487775965fb1be2a"
server_key_version: "encryptionkey@0"
I20260924 19:13:26.582796 19970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:13:26.586865 19985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:26.588436 19970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:13:26.588650 19970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "fe97477e6cb04b2ebfa2263e1fe9c9fd"
format_stamp: "Formatted at 2026-09-24 19:13:26 on dist-test-slave-0lgr"
server_key: "37fe6cd1492b41f3a60096ccb9265e48"
server_key_iv: "3efa40a6b065d719487775965fb1be2a"
server_key_version: "encryptionkey@0"
I20260924 19:13:26.588887 19970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:26.596876 19830 heartbeater.cc:503] Master 127.11.18.126:45179 was elected leader, sending a full tablet report...
I20260924 19:13:26.624766 19970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:26.629289 19970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:26.629611 19970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:26.630662 19970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:26.631979 19970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:26.632082 19970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:26.632241 19970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:26.632313 19970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:26.667804 19970 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:42487
I20260924 19:13:26.667896 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:42487 every 8 connection(s)
I20260924 19:13:26.669263 19970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:26.670174 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 19970
I20260924 19:13:26.670367 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:13:26.671047 11337 external_mini_cluster.cc:1428] Setting key 1dd446fb63016bd98c2abce6930c7462
Sep 24 19:13:26 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277206, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:26.691964 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:26.672410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:52619 (local address 127.11.18.126:45179)
0924 19:13:26.672984 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:26.673002 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:26.674147 (+ 1145us) server_negotiation.cc:407] Connection header received
0924 19:13:26.676337 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:26.676368 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:26.676470 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:26.676599 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:26.678293 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.679157 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.680961 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.681473 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.681627 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:26.685555 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:26.685561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:26.685564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:26.685597 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:26.688564 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:26.689347 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:26.689354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:26.689358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:26.689483 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:26.689938 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:26.689943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:26.689947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:26.690231 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:26.690396 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:26.691271 (+ 875us) server_negotiation.cc:299] Negotiation successful
0924 19:13:26.691515 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":322,"threads_started":1}
I20260924 19:13:26.693042 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:26.673074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:52619)
0924 19:13:26.673990 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:26.674032 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:26.676040 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:26.676869 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:26.676878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:26.677501 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:26.678118 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:26.678137 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.679337 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.679342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:26.680794 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:26.680806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.681723 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.681732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:26.681985 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:26.682686 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:26.682711 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:26.685265 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:26.688715 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:26.688724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:26.688742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:26.689188 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:26.689603 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:26.689609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:26.689613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:26.689812 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:26.690368 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:26.690377 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:26.690782 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 19:13:26.691167 (+ 385us) client_negotiation.cc:240] Negotiation successful
0924 19:13:26.691767 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":432,"threads_started":1}
I20260924 19:13:26.694773 20098 heartbeater.cc:347] Connected to a master server at 127.11.18.126:45179
I20260924 19:13:26.695128 20098 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:26.696026 20098 heartbeater.cc:511] Master 127.11.18.126:45179 requested a full tablet report, sending...
I20260924 19:13:26.697607 19641 ts_manager.cc:194] Registered new tserver with Master: fe97477e6cb04b2ebfa2263e1fe9c9fd (127.11.18.67:42487)
I20260924 19:13:26.698884 19641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:52619
I20260924 19:13:26.700124 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab.
Entry for principal oryx/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab.
WARNING: no policy specified for oryx/127.11.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab.
Entry for principal oryx/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab.
WARNING: no policy specified for oryx/127.11.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab.
Entry for principal oryx/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab.
I20260924 19:13:26.866006 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 19702
Sep 24 19:13:26 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:13:26.884213 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:26.867376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.11.18.65:40955)
0924 19:13:26.868038 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:26.868047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:26.868086 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:13:26.868149 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:26.868156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:26.868388 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:26.868534 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:26.869958 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.871074 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.872554 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.873159 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.873260 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:26.878032 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:26.878046 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:26.878062 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:26.878098 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:26.881066 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:26.881987 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:26.881995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:26.881997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:26.882123 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:26.882572 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:26.882575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:26.882577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:26.882925 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:26.883154 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:26.883611 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 19:13:26.883872 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":339,"threads_started":1}
I20260924 19:13:26.885725 19825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:26.975768 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 19702
I20260924 19:13:26.992511 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 19836
Sep 24 19:13:26 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:13:27.008726 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:26.993411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.11.18.66:46353)
0924 19:13:26.993947 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:26.993954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:26.993979 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:13:26.994035 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:26.994042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:26.994321 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:26.994498 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:26.995745 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.996744 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.998331 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:26.998864 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:26.999038 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:27.003362 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:27.003375 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:27.003389 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:27.003424 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:27.006148 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:27.006828 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:27.006835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:27.006837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:27.006950 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:27.007350 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:27.007354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:27.007356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:27.007691 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:27.007918 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:27.008224 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 19:13:27.008419 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":257,"threads_started":1}
I20260924 19:13:27.010028 19959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:27.094365 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 19836
I20260924 19:13:27.111582 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 19970
Sep 24 19:13:27 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:13:27.129375 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:27.112844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.11.18.67:42487)
0924 19:13:27.113465 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:27.113475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:27.113508 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:13:27.113572 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:27.113578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:27.113784 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:27.113980 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:27.115274 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:27.116253 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:27.117630 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:27.118184 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:27.118324 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:27.122916 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:27.122928 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:27.122943 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:27.122988 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:27.126215 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:27.127059 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:27.127067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:27.127070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:27.127204 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:27.127597 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:27.127604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:27.127608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:27.127986 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:27.128305 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:27.128642 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 19:13:27.128922 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":274,"threads_started":1}
I20260924 19:13:27.130872 20093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:27.219314 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 19970
I20260924 19:13:27.235396 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 19609
I20260924 19:13:27.236755 19671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:27.338693 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 19609
I20260924 19:13:27.359100 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45179
--webserver_interface=127.11.18.126
--webserver_port=46305
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:27.696543 20116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:27.696879 20116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:27.697022 20116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:27.705194 20116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:27.705361 20116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:27.705458 20116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:27.705561 20116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:27.705675 20116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:27.716027 20116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=46305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:27.717476 20116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:27.718827 20116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:27.730640 20121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:27.730612 20124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:27.730723 20122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:27.732694 20116 server_base.cc:1034] running on GCE node
I20260924 19:13:27.733563 20116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:27.734984 20116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:27.736272 20116 hybrid_clock.cc:648] HybridClock initialized: now 1790277207736173 us; error 55 us; skew 500 ppm
I20260924 19:13:27.736677 20116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:27 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277207, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:27.740358 20116 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:13:27.742168 20116 webserver.cc:460] Webserver started at http://127.11.18.126:46305/ using document root <none> and password file <none>
I20260924 19:13:27.743068 20116 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:27.743238 20116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:27.749076 20116 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:13:27.752322 20131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:27.754016 20116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:13:27.754273 20116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "f21ba721014940a5992be9eabcfa5691"
format_stamp: "Formatted at 2026-09-24 19:13:24 on dist-test-slave-0lgr"
server_key: "82176ba25133c4214d9a8372dbc16a0f"
server_key_iv: "b6b90414849b7d41f4e68d39294b4bc2"
server_key_version: "encryptionkey@0"
I20260924 19:13:27.755272 20116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:27.782732 20116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:27.787590 20116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:27.787884 20116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:27.807998 20116 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:45179
I20260924 19:13:27.808089 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:45179 every 8 connection(s)
I20260924 19:13:27.809568 20116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:13:27.812520 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 20116
I20260924 19:13:27.813553 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:40955
--local_ip_for_outbound_sockets=127.11.18.65
--tserver_master_addrs=127.11.18.126:45179
--webserver_port=32811
--webserver_interface=127.11.18.65
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.18.65
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:13:27.818774 20183 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:13:27.823259 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Bootstrap starting.
I20260924 19:13:27.842659 20183 log.cc:824] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:27.851218 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:13:27.851748 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Bootstrap complete.
I20260924 19:13:27.861147 20183 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } }
I20260924 19:13:27.862023 20183 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21ba721014940a5992be9eabcfa5691, State: Initialized, Role: FOLLOWER
I20260924 19:13:27.862679 20183 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } }
I20260924 19:13:27.862906 20183 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:13:27.863016 20183 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:13:27.863190 20183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:13:27.867054 20183 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } }
I20260924 19:13:27.867578 20183 leader_election.cc:302] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21ba721014940a5992be9eabcfa5691; no voters:
I20260924 19:13:27.868261 20183 leader_election.cc:288] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:13:27.868399 20187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:13:27.869621 20187 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [term 2 LEADER]: Becoming Leader. State: Replica: f21ba721014940a5992be9eabcfa5691, State: Running, Role: LEADER
I20260924 19:13:27.870195 20187 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } }
I20260924 19:13:27.870592 20183 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:27.873203 20188 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f21ba721014940a5992be9eabcfa5691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } } }
I20260924 19:13:27.873540 20188 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:27.872850 20189 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21ba721014940a5992be9eabcfa5691. Latest consensus state: current_term: 2 leader_uuid: "f21ba721014940a5992be9eabcfa5691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ba721014940a5992be9eabcfa5691" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45179 } } }
I20260924 19:13:27.874075 20189 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:27.879081 20193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:13:27.886348 20193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:13:27.888118 20193 catalog_manager.cc:1259] Loaded cluster ID: 25d1775e02c942e7b12de81f69827a61
I20260924 19:13:27.888235 20193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:13:27.893369 20193 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:13:27.895092 20193 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f21ba721014940a5992be9eabcfa5691: Loaded TSK: 0
I20260924 19:13:27.897022 20193 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:13:28.178853 20185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:28.179214 20185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:28.179365 20185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:28.187381 20185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:28.187536 20185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:28.187697 20185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:28.197818 20185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.65.keytab
--principal=oryx/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:40955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=32811
--tserver_master_addrs=127.11.18.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:28.199173 20185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:28.200443 20185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:28.216357 20210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:28.216297 20212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:28.216981 20209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:28.217406 20185 server_base.cc:1034] running on GCE node
I20260924 19:13:28.218035 20185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:28.219022 20185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:28.220270 20185 hybrid_clock.cc:648] HybridClock initialized: now 1790277208220203 us; error 55 us; skew 500 ppm
I20260924 19:13:28.220652 20185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:28 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277208, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:28.224267 20185 init.cc:377] Logged in from keytab as oryx/127.11.18.65@KRBTEST.COM (short username oryx)
I20260924 19:13:28.226058 20185 webserver.cc:460] Webserver started at http://127.11.18.65:32811/ using document root <none> and password file <none>
I20260924 19:13:28.226819 20185 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:28.226981 20185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:28.232944 20185 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 19:13:28.235985 20219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:28.237588 20185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:13:28.237776 20185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "5fbbc292916341999b4f88847f11343d"
format_stamp: "Formatted at 2026-09-24 19:13:25 on dist-test-slave-0lgr"
server_key: "8741c0b69f7744ed0905b25d476c5459"
server_key_iv: "c0f43f1b50fe2b44316057012737cd8d"
server_key_version: "encryptionkey@0"
I20260924 19:13:28.239310 20185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:28.277966 20185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:28.282644 20185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:28.282938 20185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:28.283924 20185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:28.285270 20185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:28.285416 20185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:28.285535 20185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:28.285634 20185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:28.316123 20185 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:40955
I20260924 19:13:28.316210 20331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:40955 every 8 connection(s)
I20260924 19:13:28.317691 20185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:28.319439 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 20185
I20260924 19:13:28.320159 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:46353
--local_ip_for_outbound_sockets=127.11.18.66
--tserver_master_addrs=127.11.18.126:45179
--webserver_port=37923
--webserver_interface=127.11.18.66
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.18.66
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:13:28 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:28 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:13:28.663942 20336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:28.664225 20336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:28.664364 20336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:28.672125 20336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:28.672243 20336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:28.672447 20336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:28.682857 20336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.66.keytab
--principal=oryx/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:46353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=37923
--tserver_master_addrs=127.11.18.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:28.684222 20336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:28.685613 20336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:28.701705 20342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:28.701745 20343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:28.702870 20345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:28.707861 20336 server_base.cc:1034] running on GCE node
I20260924 19:13:28.708730 20336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:28.709939 20336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:28.711199 20336 hybrid_clock.cc:648] HybridClock initialized: now 1790277208711134 us; error 52 us; skew 500 ppm
I20260924 19:13:28.711598 20336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:28 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277208, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:28.715417 20336 init.cc:377] Logged in from keytab as oryx/127.11.18.66@KRBTEST.COM (short username oryx)
I20260924 19:13:28.717389 20336 webserver.cc:460] Webserver started at http://127.11.18.66:37923/ using document root <none> and password file <none>
I20260924 19:13:28.718159 20336 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:28.718313 20336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:28.723928 20336 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 19:13:28.726794 20352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:28.728240 20336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:13:28.728410 20336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "cf7720bb3bdb4e51b6619f961157f87e"
format_stamp: "Formatted at 2026-09-24 19:13:26 on dist-test-slave-0lgr"
server_key: "f93c30aeb1ba7309083b89e4f1d93ecd"
server_key_iv: "045eed3ff9343848ae592724ce597d90"
server_key_version: "encryptionkey@0"
I20260924 19:13:28.729398 20336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:28.750052 20336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:28.754653 20336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:28.754930 20336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:28.756027 20336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:28.757589 20336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:28.757700 20336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:28.757828 20336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:28.757912 20336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:28.801461 20336 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:46353
I20260924 19:13:28.801503 20464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:46353 every 8 connection(s)
I20260924 19:13:28.802868 20336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:28.804786 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 20336
I20260924 19:13:28.805821 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:42487
--local_ip_for_outbound_sockets=127.11.18.67
--tserver_master_addrs=127.11.18.126:45179
--webserver_port=42369
--webserver_interface=127.11.18.67
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.18.67
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:13:28 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:28 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:13:29.180294 20469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:29.180639 20469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:29.180819 20469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:29.189472 20469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:29.189704 20469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:29.189920 20469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:29.200954 20469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/krb5kdc/oryx_127.11.18.67.keytab
--principal=oryx/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:42487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=42369
--tserver_master_addrs=127.11.18.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:29.202478 20469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:29.203905 20469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:29.226567 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:29.227661 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:29.230077 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:29.231516 20469 server_base.cc:1034] running on GCE node
I20260924 19:13:29.232213 20469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:29.233716 20469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:29.235019 20469 hybrid_clock.cc:648] HybridClock initialized: now 1790277209234937 us; error 67 us; skew 500 ppm
I20260924 19:13:29.235386 20469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:29 dist-test-slave-0lgr krb5kdc[19596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277209, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:29.239167 20469 init.cc:377] Logged in from keytab as oryx/127.11.18.67@KRBTEST.COM (short username oryx)
I20260924 19:13:29.241088 20469 webserver.cc:460] Webserver started at http://127.11.18.67:42369/ using document root <none> and password file <none>
I20260924 19:13:29.241983 20469 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:29.242136 20469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:29.248049 20469 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:13:29.250881 20485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:29.252311 20469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:13:29.252488 20469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "fe97477e6cb04b2ebfa2263e1fe9c9fd"
format_stamp: "Formatted at 2026-09-24 19:13:26 on dist-test-slave-0lgr"
server_key: "37fe6cd1492b41f3a60096ccb9265e48"
server_key_iv: "3efa40a6b065d719487775965fb1be2a"
server_key_version: "encryptionkey@0"
I20260924 19:13:29.253474 20469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:29.285605 20469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:29.290241 20469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:29.290522 20469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:29.291599 20469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:29.293079 20469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:29.293234 20469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:29.293402 20469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:29.293514 20469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:29.362110 20469 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:42487
I20260924 19:13:29.362210 20597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:42487 every 8 connection(s)
I20260924 19:13:29.365003 20469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:29.372211 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 20469
Sep 24 19:13:29 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:29 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:29.455253 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:29.374299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.11.18.126:45179)
0924 19:13:29.423142 (+ 48843us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:29.423152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:29.423176 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:13:29.423261 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:29.423271 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:29.423401 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:29.423574 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:29.427111 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.428895 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.434250 (+ 5355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.435200 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.435374 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:29.442124 (+ 6750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:29.442141 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:29.442172 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:29.442246 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:29.448296 (+ 6050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:29.450162 (+ 1866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:29.450177 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:29.450193 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:29.450614 (+ 421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:29.451247 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:29.451255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:29.451261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:29.452132 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:29.452464 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:29.453649 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0924 19:13:29.454125 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48665,"thread_start_us":43010,"threads_started":1}
W20260924 19:13:29.676647 20334 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.344s user 0.001s sys 0.012s
W20260924 19:13:29.678109 20335 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:28.321998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:58631 (local address 127.11.18.126:45179)
0924 19:13:28.322742 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:28.322760 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:28.323261 (+ 501us) server_negotiation.cc:407] Connection header received
0924 19:13:28.325482 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:28.325496 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:28.325863 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:28.326234 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:28.327810 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:28.329015 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:28.330616 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:28.331269 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:28.331470 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:29.677775 (+1346305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:58631: BlockingRecv error: recv got EOF from 127.11.18.65:58631 (error 108)
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":292,"threads_started":1}
I20260924 19:13:29.678741 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:28.321229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:58631)
0924 19:13:28.323072 (+ 1843us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:28.323112 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:28.324854 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:28.326422 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:28.326436 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:28.327071 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:28.327601 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:28.327628 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:28.329182 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:28.329192 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:28.330460 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:28.330472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:28.331513 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:28.331520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:28.331820 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:28.332570 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:28.332592 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:29.677542 (+1344950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1577,"thread_start_us":1315,"threads_started":1}
W20260924 19:13:29.679733 20332 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:45179 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.18.126:45179: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Sep 24 19:13:29 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:29 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:29.690758 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:29.680429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:47509)
0924 19:13:29.680689 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:29.680713 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:29.680850 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:29.681187 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:29.681191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:29.681601 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:29.681762 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:29.681770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.682788 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.682793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:29.683766 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:29.683777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.684522 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.684528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:29.684692 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:29.685523 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:29.685549 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:29.690418 (+ 4869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151}
W20260924 19:13:29.690923 20335 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:29.680462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:47509 (local address 127.11.18.126:45179)
0924 19:13:29.680624 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:29.680630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:29.680780 (+ 150us) server_negotiation.cc:407] Connection header received
0924 19:13:29.680994 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:29.680998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:29.681062 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:29.681158 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:29.681899 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.682652 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.683912 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.684313 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.684450 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:29.690642 (+ 6192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:47509: BlockingRecv error: recv got EOF from 127.11.18.65:47509 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 19:13:29 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:29 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:29.702507 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:29.691772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:55295)
0924 19:13:29.692099 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:29.692114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:29.692256 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:29.692633 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:29.692640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:29.692981 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:29.693142 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:29.693151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.694274 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.694281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:29.695347 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:29.695359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.696337 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.696343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:29.696560 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:29.697377 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:29.697445 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:29.702171 (+ 4726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240}
W20260924 19:13:29.702677 20335 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:29.691835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:55295 (local address 127.11.18.126:45179)
0924 19:13:29.691994 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:29.691998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:29.692188 (+ 190us) server_negotiation.cc:407] Connection header received
0924 19:13:29.692408 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:29.692414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:29.692492 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:29.692598 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:29.693283 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.694124 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.695507 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.696162 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.696304 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:29.702374 (+ 6070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:55295: BlockingRecv error: recv got EOF from 127.11.18.65:55295 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 19:13:29.703449 20332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:13:30.299263 20468 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.466s user 0.004s sys 0.014s
W20260924 19:13:30.300613 20467 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:28.805544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:47589 (local address 127.11.18.126:45179)
0924 19:13:28.822057 (+ 16513us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:28.822063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:28.822090 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:13:28.824004 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:28.824013 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:28.824132 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:28.824238 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:28.826708 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:28.828050 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:28.830239 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:28.831096 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:28.831236 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.300342 (+1469106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:47589: BlockingRecv error: recv got EOF from 127.11.18.66:47589 (error 108)
Metrics: {"server-negotiator.queue_time_us":16360,"thread_start_us":16556,"threads_started":1}
I20260924 19:13:30.301151 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:28.806359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:47589)
0924 19:13:28.820744 (+ 14385us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:28.820833 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:28.823572 (+ 2739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:28.824737 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:28.824753 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:28.825624 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:28.826460 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:28.826489 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:28.828280 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:28.828288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:28.830060 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:28.830078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:28.831588 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:28.831597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:28.831930 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:28.832727 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:28.832764 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:30.300068 (+1467304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13879,"thread_start_us":15705,"threads_started":1}
W20260924 19:13:30.302372 20465 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:45179 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.18.126:45179: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:30.317952 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:30.303238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:49421)
0924 19:13:30.303413 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:30.303432 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:30.303530 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:30.303984 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:30.303988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:30.304428 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:30.304595 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.304603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.305652 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.305658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.306721 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.306732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.307408 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.307413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.307579 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.312492 (+ 4913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:30.312512 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:30.317611 (+ 5099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 19:13:30.318341 20467 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:30.303501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:49421 (local address 127.11.18.126:45179)
0924 19:13:30.303695 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:30.303702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:30.303719 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:30.303775 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:30.303779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:30.303854 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:30.303952 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:30.304726 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.305520 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.306851 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.307245 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.307380 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.317995 (+ 10615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:49421: BlockingRecv error: recv got EOF from 127.11.18.66:49421 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:30.330335 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:30.319650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:49813)
0924 19:13:30.319801 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:30.319818 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:30.319908 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:30.320491 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:30.320496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:30.320864 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:30.321078 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.321088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.322143 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.322148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.323108 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.323118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.323823 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.323828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.324020 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.324731 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:30.324747 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:30.329913 (+ 5166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 19:13:30.330862 20465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:13:30.330592 20467 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:30.319972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:49813 (local address 127.11.18.126:45179)
0924 19:13:30.320178 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:30.320182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:30.320200 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:13:30.320242 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:30.320245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:30.320320 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:30.320508 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:30.321217 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.322002 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.323239 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.323664 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.323808 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.330128 (+ 6320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:49813: BlockingRecv error: recv got EOF from 127.11.18.66:49813 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:30.715031 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:30.704041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:46359)
0924 19:13:30.704686 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:30.704709 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:30.704818 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:30.705175 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:30.705178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:30.705623 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:30.705762 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.705770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.707191 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.707196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.708263 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.708276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.709490 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.709495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.709626 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.710305 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:30.710324 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:30.714626 (+ 4302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":407,"threads_started":1}
W20260924 19:13:30.715106 20467 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:30.704199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:46359 (local address 127.11.18.126:45179)
0924 19:13:30.704399 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:30.704406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:30.704783 (+ 377us) server_negotiation.cc:407] Connection header received
0924 19:13:30.704922 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:30.704928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:30.705030 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:30.705139 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:30.705899 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.706990 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.708515 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.709235 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.709405 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.714841 (+ 5436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:46359: BlockingRecv error: recv got EOF from 127.11.18.65:46359 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 19:13:30.725891 20601 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.276s user 0.001s sys 0.023s
W20260924 19:13:30.727403 20603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:29.369698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:32943 (local address 127.11.18.126:45179)
0924 19:13:29.416982 (+ 47284us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:29.416993 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:29.422651 (+ 5658us) server_negotiation.cc:407] Connection header received
0924 19:13:29.436707 (+ 14056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:29.436717 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:29.436852 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:29.437053 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:29.440715 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.442530 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.446378 (+ 3848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.447552 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.447706 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.727019 (+1279313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:32943: BlockingRecv error: recv got EOF from 127.11.18.67:32943 (error 108)
Metrics: {"server-negotiator.queue_time_us":47093,"thread_start_us":59356,"threads_started":1}
I20260924 19:13:30.728014 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:29.371070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:32943)
0924 19:13:29.418935 (+ 47865us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:29.419016 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:29.431032 (+ 12016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:29.437822 (+ 6790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:29.437842 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:29.439105 (+ 1263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:29.440363 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:29.440401 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.442879 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.442889 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:29.446095 (+ 3206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:29.446122 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:29.448315 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:29.448325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:29.448777 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:29.449881 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:29.449929 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:30.726819 (+1276890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47337,"thread_start_us":50097,"threads_started":1}
W20260924 19:13:30.728842 20598 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:45179 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.18.126:45179: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:30.738171 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:30.729457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:36077)
0924 19:13:30.729665 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:30.729687 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:30.729795 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:30.730109 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:30.730113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:30.730490 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:30.730664 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.730673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.731728 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.731735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.732868 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.732880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.733628 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.733634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.733800 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.734447 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:30.734465 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:30.737806 (+ 3341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 19:13:30.738214 20603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:30.729573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:36077 (local address 127.11.18.126:45179)
0924 19:13:30.729804 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:30.729809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:30.729824 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:13:30.729899 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:30.729902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:30.729958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:30.730047 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:30.730839 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.731578 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.733004 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.733465 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.733596 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.738033 (+ 4437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:36077: BlockingRecv error: recv got EOF from 127.11.18.67:36077 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:30 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:30.747864 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:30.738874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:35875)
0924 19:13:30.739110 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:30.739129 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:30.739238 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:30.739637 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:30.739642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:30.740057 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:30.740221 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.740229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.741247 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.741251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.742264 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:30.742275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.742946 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.742950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:30.743154 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.743797 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:30.743816 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:30.747658 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260924 19:13:30.748198 20598 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:13:30.748324 20603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:30.738997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:35875 (local address 127.11.18.126:45179)
0924 19:13:30.739143 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:30.739146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:30.739181 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:13:30.739379 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:30.739384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:30.739488 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:30.739589 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:30.740352 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.741118 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.742398 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:30.742808 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:30.742908 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:30.747887 (+ 4979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:35875: BlockingRecv error: recv got EOF from 127.11.18.67:35875 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 19:13:31 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:31 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:31.341759 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:31.331473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:44289)
0924 19:13:31.332118 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:31.332135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:31.332250 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:31.332694 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:31.332699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:31.333292 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:31.333524 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:31.333535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.334608 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.334612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:31.335693 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:31.335705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.336657 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.336662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:31.336842 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:31.337646 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:31.337666 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:31.341382 (+ 3716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":405,"threads_started":1}
W20260924 19:13:31.341893 20607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:31.331630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:44289 (local address 127.11.18.126:45179)
0924 19:13:31.332124 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:31.332130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:31.332190 (+ 60us) server_negotiation.cc:407] Connection header received
0924 19:13:31.332428 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:31.332433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:31.332503 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:31.332615 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:31.333676 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.334464 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.335869 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.336465 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.336619 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:31.341667 (+ 5048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:44289: BlockingRecv error: recv got EOF from 127.11.18.66:44289 (error 108)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
Sep 24 19:13:31 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:31 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:31.726557 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:31.716292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:36049)
0924 19:13:31.716957 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:31.716978 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:31.717106 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:31.717529 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:31.717533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:31.718063 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:31.718259 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:31.718268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.719288 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.719292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:31.720316 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:31.720329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.721224 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.721229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:31.721434 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:31.722067 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:31.722086 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:31.726307 (+ 4221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":333,"threads_started":1}
W20260924 19:13:31.726779 20607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:31.716703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:36049 (local address 127.11.18.126:45179)
0924 19:13:31.716928 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:31.716935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:31.717062 (+ 127us) server_negotiation.cc:407] Connection header received
0924 19:13:31.717249 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:31.717253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:31.717317 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:31.717425 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:31.718399 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.719154 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.720491 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.720963 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.721102 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:31.726536 (+ 5434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:36049: BlockingRecv error: recv got EOF from 127.11.18.65:36049 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 19:13:31 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:31 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:31.758426 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:31.748933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:59677)
0924 19:13:31.749561 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:31.749577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:31.749726 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:31.750054 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:31.750059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:31.750511 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:31.750689 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:31.750697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.751705 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.751712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:31.752653 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:31.752663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.753456 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.753462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:31.753617 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:31.754223 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:31.754240 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:31.758091 (+ 3851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":321,"threads_started":1}
W20260924 19:13:31.758633 20607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:31.749218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:59677 (local address 127.11.18.126:45179)
0924 19:13:31.749457 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:31.749463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:31.749656 (+ 193us) server_negotiation.cc:407] Connection header received
0924 19:13:31.749858 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:31.749868 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:31.749927 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:31.750021 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:31.750827 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.751576 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.752786 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:31.753226 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:31.753375 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:31.758323 (+ 4948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:59677: BlockingRecv error: recv got EOF from 127.11.18.67:59677 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 19:13:32 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:32 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:32.352831 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:32.342758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:41793)
0924 19:13:32.343618 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:32.343636 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:32.343755 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:32.344210 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:32.344213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:32.344641 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:32.344819 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:32.344828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.345914 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.345918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:32.346910 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:32.346922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.347687 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.347691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:32.347815 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:32.348746 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:32.348764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:32.352472 (+ 3708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":609,"threads_started":1}
W20260924 19:13:32.353056 20611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:32.342830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:41793 (local address 127.11.18.126:45179)
0924 19:13:32.343307 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:32.343312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:32.343708 (+ 396us) server_negotiation.cc:407] Connection header received
0924 19:13:32.343872 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:32.343926 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:32.344064 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:32.344167 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:32.344942 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.345779 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.347066 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.347519 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.347690 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:32.352696 (+ 5006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:41793: BlockingRecv error: recv got EOF from 127.11.18.66:41793 (error 108)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":276,"threads_started":1}
Sep 24 19:13:32 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:32 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:32.738528 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:32.727839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:60399)
0924 19:13:32.728563 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:32.728580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:32.728763 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:32.729296 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:32.729301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:32.729878 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:32.730059 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:32.730068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.731414 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.731418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:32.732531 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:32.732544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.733310 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.733315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:32.733554 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:32.734254 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:32.734273 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:32.738261 (+ 3988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":342,"threads_started":1}
W20260924 19:13:32.738983 20611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:32.728227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:60399 (local address 127.11.18.126:45179)
0924 19:13:32.728476 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:32.728493 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:32.728675 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:13:32.729046 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:32.729051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:32.729118 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:32.729219 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:32.730217 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.731232 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.732677 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.733147 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.733263 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:32.738538 (+ 5275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:60399: BlockingRecv error: recv got EOF from 127.11.18.65:60399 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 19:13:32 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:32 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:32.769656 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:32.759377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:60741)
0924 19:13:32.759935 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:32.759952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:32.760106 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:32.760537 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:32.760540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:32.760943 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:32.761131 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:32.761140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.762413 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.762418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:32.763407 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:32.763421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.764219 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.764224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:32.764369 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:32.765068 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:32.765087 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:32.769418 (+ 4331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":323,"threads_started":1}
W20260924 19:13:32.770012 20611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:32.759468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:60741 (local address 127.11.18.126:45179)
0924 19:13:32.759656 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:32.759661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:32.760060 (+ 399us) server_negotiation.cc:407] Connection header received
0924 19:13:32.760232 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:32.760237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:32.760322 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:32.760461 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:32.761303 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.762237 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.763576 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:32.764043 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:32.764154 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:32.769592 (+ 5438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:60741: BlockingRecv error: recv got EOF from 127.11.18.67:60741 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 19:13:33 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:33 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:33.363306 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:33.353922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:55953)
0924 19:13:33.354495 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:33.354514 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:33.354658 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:33.355138 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:33.355143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:33.355500 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:33.355660 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:33.355668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.356688 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.356693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:33.357857 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:33.357869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.358576 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.358580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:33.358699 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:33.359298 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:33.359316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:33.363081 (+ 3765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
W20260924 19:13:33.363663 20615 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:33.354056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:55953 (local address 127.11.18.126:45179)
0924 19:13:33.354550 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:33.354556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:33.354575 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:13:33.354807 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:33.354815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:33.354906 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:33.355019 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:33.355798 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.356545 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.357998 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.358452 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.358560 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:33.363367 (+ 4807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:55953: BlockingRecv error: recv got EOF from 127.11.18.66:55953 (error 108)
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":287,"threads_started":1}
Sep 24 19:13:33 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:33 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:33.749900 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:33.739584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:51411)
0924 19:13:33.740110 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:33.740133 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:33.740260 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:33.740639 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:33.740643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:33.741052 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:33.741221 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:33.741230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.742589 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.742598 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:33.743946 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:33.743963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.744741 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.744745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:33.744943 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:33.745614 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:33.745633 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:33.749613 (+ 3980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":319,"threads_started":1}
W20260924 19:13:33.750097 20615 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:33.739741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:51411 (local address 127.11.18.126:45179)
0924 19:13:33.739928 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:33.739933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:33.740242 (+ 309us) server_negotiation.cc:407] Connection header received
0924 19:13:33.740399 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:33.740404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:33.740470 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:33.740574 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:33.741454 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.742423 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.744113 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.744578 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.744681 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:33.749853 (+ 5172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:51411: BlockingRecv error: recv got EOF from 127.11.18.65:51411 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 19:13:33 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:33 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:33.779896 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:33.770655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:40147)
0924 19:13:33.771121 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:33.771136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:33.771292 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:33.771726 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:33.771731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:33.772088 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:33.772244 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:33.772252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.773268 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.773273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:33.774225 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:33.774236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.774929 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.774935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:33.775119 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:33.775730 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:33.775747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:33.779457 (+ 3710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":237,"threads_started":1}
W20260924 19:13:33.780213 20615 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:33.770692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:40147 (local address 127.11.18.126:45179)
0924 19:13:33.770884 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:33.770891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:33.771217 (+ 326us) server_negotiation.cc:407] Connection header received
0924 19:13:33.771457 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:33.771462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:33.771548 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:33.771647 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:33.772383 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.773126 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.774377 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:33.774787 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:33.774915 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:33.779737 (+ 4822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:40147: BlockingRecv error: recv got EOF from 127.11.18.67:40147 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 19:13:34 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:34 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:34.374621 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:34.364531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:54183)
0924 19:13:34.365108 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:34.365125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:34.365230 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:34.366002 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:34.366010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:34.366456 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:34.366665 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:34.366674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.367862 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.367866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:34.368930 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:34.368942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.369733 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.369737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:34.369949 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:34.370676 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:34.370695 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:34.374360 (+ 3665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":314,"threads_started":1}
W20260924 19:13:34.374971 20619 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:34.364621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:54183 (local address 127.11.18.126:45179)
0924 19:13:34.365541 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:34.365548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:34.365565 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:34.365618 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:34.365625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:34.365723 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:34.365857 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:34.366855 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.367741 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.369081 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.369579 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.369691 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:34.374630 (+ 4939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:54183: BlockingRecv error: recv got EOF from 127.11.18.66:54183 (error 108)
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":693,"threads_started":1}
Sep 24 19:13:34 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:34 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:34.761407 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:34.750901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:58051)
0924 19:13:34.751640 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:34.751657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:34.751781 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:34.752213 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:34.752218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:34.752678 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:34.752910 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:34.752926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.754442 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.754449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:34.755524 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:34.755537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.756298 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.756304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:34.756432 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:34.757175 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:34.757193 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:34.761039 (+ 3846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":401,"threads_started":1}
W20260924 19:13:34.761790 20619 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:34.751108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:58051 (local address 127.11.18.126:45179)
0924 19:13:34.751302 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:34.751307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:34.751742 (+ 435us) server_negotiation.cc:407] Connection header received
0924 19:13:34.751885 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:34.751889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:34.751956 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:34.752062 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:34.753128 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.754282 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.755667 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.756138 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.756269 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:34.761244 (+ 4975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:58051: BlockingRecv error: recv got EOF from 127.11.18.65:58051 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 19:13:34 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:34 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:34.791033 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:34.780902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:55209)
0924 19:13:34.781489 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:34.781506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:34.781663 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:34.782131 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:34.782137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:34.782607 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:34.782772 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:34.782782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.783861 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.783869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:34.785003 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:34.785014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.785827 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.785834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:34.786039 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:34.786669 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:34.786689 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:34.790681 (+ 3992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":306,"threads_started":1}
W20260924 19:13:34.791615 20619 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:34.781069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:55209 (local address 127.11.18.126:45179)
0924 19:13:34.781349 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:34.781358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:34.781589 (+ 231us) server_negotiation.cc:407] Connection header received
0924 19:13:34.781890 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:34.781894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:34.781962 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:34.782088 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:34.782916 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.783718 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.785146 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:34.785671 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:34.785822 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:34.791134 (+ 5312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:55209: BlockingRecv error: recv got EOF from 127.11.18.67:55209 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 19:13:35 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:35 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:35.393015 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:35.375664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:42767)
0924 19:13:35.376167 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:35.376184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:35.376289 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:35.376722 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:35.376728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:35.377232 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:35.377499 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:35.377510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.378768 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.378775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:35.379844 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:35.379857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.380550 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.380556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:35.380735 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:35.388507 (+ 7772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:35.388530 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:35.392770 (+ 4240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":293,"threads_started":1}
W20260924 19:13:35.393307 20623 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:35.375909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:42767 (local address 127.11.18.126:45179)
0924 19:13:35.376395 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:35.376400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:35.376416 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:13:35.376473 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:35.376477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:35.376538 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:35.376637 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:35.377687 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.378642 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.379978 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.380401 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.380535 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:35.393041 (+ 12506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:42767: BlockingRecv error: recv got EOF from 127.11.18.66:42767 (error 108)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":260,"threads_started":1}
Sep 24 19:13:35 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:35 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:35.772902 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:35.762346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:48879)
0924 19:13:35.762905 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:35.762932 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:35.763060 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:35.763387 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:35.763391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:35.763918 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:35.764096 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:35.764106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.765252 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.765257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:35.766405 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:35.766421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.767193 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.767199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:35.767363 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:35.768127 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:35.768147 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:35.772511 (+ 4364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":357,"threads_started":1}
W20260924 19:13:35.772962 20623 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:35.762533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:48879 (local address 127.11.18.126:45179)
0924 19:13:35.762804 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:35.762811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:35.763079 (+ 268us) server_negotiation.cc:407] Connection header received
0924 19:13:35.763145 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:35.763149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:35.763218 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:35.763313 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:35.764276 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.765087 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.766563 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.767019 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.767119 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:35.772724 (+ 5605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:48879: BlockingRecv error: recv got EOF from 127.11.18.65:48879 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 24 19:13:35 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:35 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:35.802664 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:35.792138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:45583)
0924 19:13:35.792753 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:35.792770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:35.792896 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:35.793231 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:35.793235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:35.793714 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:35.793869 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:35.793878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.795283 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.795291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:35.796582 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:35.796596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.797496 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.797501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:35.797632 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:35.798323 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:35.798342 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:35.802316 (+ 3974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":347,"threads_started":1}
W20260924 19:13:35.802863 20623 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:35.792230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:45583 (local address 127.11.18.126:45179)
0924 19:13:35.792428 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:35.792433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:35.792857 (+ 424us) server_negotiation.cc:407] Connection header received
0924 19:13:35.793024 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:35.793028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:35.793095 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:35.793192 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:35.794053 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.795075 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.796743 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:35.797242 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:35.797470 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:35.802573 (+ 5103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:45583: BlockingRecv error: recv got EOF from 127.11.18.67:45583 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 19:13:36 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:36 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:36.404356 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:36.394062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:36397)
0924 19:13:36.394544 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:36.394562 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:36.394673 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:36.395253 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:36.395257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:36.395656 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:36.395900 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:36.395911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.397148 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.397154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:36.398547 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:36.398566 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.399357 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.399363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:36.399503 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:36.400169 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:36.400188 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:36.404112 (+ 3924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":271,"threads_started":1}
W20260924 19:13:36.404830 20627 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:36.394180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:36397 (local address 127.11.18.126:45179)
0924 19:13:36.394915 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:36.394920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:36.394936 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:13:36.394988 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:36.394992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:36.395062 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:36.395173 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:36.396065 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.396937 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.398743 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.399194 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.399300 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:36.404378 (+ 5078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:36397: BlockingRecv error: recv got EOF from 127.11.18.66:36397 (error 108)
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":533,"threads_started":1}
Sep 24 19:13:36 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:36 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:36.783351 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:36.773924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:56851)
0924 19:13:36.774463 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:36.774479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:36.774635 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:36.774980 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:36.774986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:36.775411 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:36.775584 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:36.775592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.776608 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.776616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:36.777667 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:36.777680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.778374 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.778380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:36.778554 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:36.779141 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:36.779159 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:36.782997 (+ 3838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":240,"threads_started":1}
W20260924 19:13:36.783424 20627 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:36.773989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:56851 (local address 127.11.18.126:45179)
0924 19:13:36.774201 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:36.774209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:36.774568 (+ 359us) server_negotiation.cc:407] Connection header received
0924 19:13:36.774756 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:36.774763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:36.774837 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:36.774940 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:36.775727 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.776476 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.777816 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.778220 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.778360 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:36.783204 (+ 4844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:56851: BlockingRecv error: recv got EOF from 127.11.18.65:56851 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 19:13:36 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:36 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:36.814582 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:36.803941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:44535)
0924 19:13:36.804502 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:36.804527 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:36.804680 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:36.805046 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:36.805053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:36.805602 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:36.805769 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:36.805779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.806979 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.806989 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:36.808148 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:36.808168 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.809083 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.809091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:36.809308 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:36.810132 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:36.810151 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:36.814227 (+ 4076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":333,"threads_started":1}
W20260924 19:13:36.815141 20627 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:36.804042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:44535 (local address 127.11.18.126:45179)
0924 19:13:36.804256 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:36.804263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:36.804614 (+ 351us) server_negotiation.cc:407] Connection header received
0924 19:13:36.804838 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:36.804842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:36.804904 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:36.805005 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:36.805986 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.806810 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.808303 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:36.808913 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:36.809047 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:36.814532 (+ 5485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:44535: BlockingRecv error: recv got EOF from 127.11.18.67:44535 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 19:13:37 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:37 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:37.415437 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:37.405414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:50633)
0924 19:13:37.406314 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:37.406339 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:37.406470 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:37.406828 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:37.406833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:37.407274 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:37.407450 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:37.407466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.408514 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.408520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:37.409569 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:37.409582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.410303 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.410309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:37.410478 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:37.411157 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:37.411177 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:37.415028 (+ 3851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":286,"threads_started":1}
W20260924 19:13:37.415602 20631 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:37.405590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:50633 (local address 127.11.18.126:45179)
0924 19:13:37.406291 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:37.406296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:37.406430 (+ 134us) server_negotiation.cc:407] Connection header received
0924 19:13:37.406583 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:37.406590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:37.406667 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:37.406791 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:37.407602 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.408365 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.409708 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.410137 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.410269 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:37.415266 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:50633: BlockingRecv error: recv got EOF from 127.11.18.66:50633 (error 108)
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":496,"threads_started":1}
Sep 24 19:13:37 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:37 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:37.794296 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:37.784761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:60383)
0924 19:13:37.785401 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:37.785420 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:37.785541 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:37.785996 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:37.785999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:37.786436 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:37.786617 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:37.786626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.787858 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.787863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:37.788840 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:37.788852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.789646 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.789651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:37.789779 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:37.790481 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:37.790500 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:37.794051 (+ 3551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":340,"threads_started":1}
W20260924 19:13:37.794580 20631 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:37.784897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:60383 (local address 127.11.18.126:45179)
0924 19:13:37.785100 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:37.785104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:37.785537 (+ 433us) server_negotiation.cc:407] Connection header received
0924 19:13:37.785683 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:37.785687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:37.785754 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:37.785929 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:37.786828 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.787610 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.788997 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.789446 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.789555 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:37.794283 (+ 4728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:60383: BlockingRecv error: recv got EOF from 127.11.18.65:60383 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 19:13:37 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:37 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:37.826985 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:37.815683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:60551)
0924 19:13:37.816763 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:37.816784 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:37.816951 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:37.817420 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:37.817425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:37.817959 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:37.818174 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:37.818186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.819340 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.819345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:37.820871 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:37.820886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.821970 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.821976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:37.822149 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:37.822728 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:37.822745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:37.826623 (+ 3878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":385,"threads_started":1}
W20260924 19:13:37.827107 20631 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:37.815990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:60551 (local address 127.11.18.126:45179)
0924 19:13:37.816221 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:37.816228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:37.816961 (+ 733us) server_negotiation.cc:407] Connection header received
0924 19:13:37.817128 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:37.817133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:37.817217 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:37.817315 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:37.818342 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.819198 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.821020 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:37.821755 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:37.821932 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:37.826848 (+ 4916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:60551: BlockingRecv error: recv got EOF from 127.11.18.67:60551 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 19:13:38 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:38 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:38.426764 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:38.416476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:35907)
0924 19:13:38.417176 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:38.417193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:38.417393 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:38.417875 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:38.417879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:38.418336 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:38.418501 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:38.418510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.419687 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.419692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:38.420746 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:38.420762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.421759 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.421764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:38.421888 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:38.422524 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:38.422541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:38.426366 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":291,"threads_started":1}
W20260924 19:13:38.426810 20635 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:38.416690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:35907 (local address 127.11.18.126:45179)
0924 19:13:38.417531 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:38.417536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:38.417552 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:13:38.417596 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:38.417600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:38.417656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:38.417769 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:38.418654 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.419533 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.420912 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.421560 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.421668 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:38.426568 (+ 4900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:35907: BlockingRecv error: recv got EOF from 127.11.18.66:35907 (error 108)
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":621,"threads_started":1}
Sep 24 19:13:38 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:38 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:38.805657 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:38.795414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:40809)
0924 19:13:38.795960 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:38.795976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:38.796126 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:38.796493 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:38.796498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:38.796920 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:38.797083 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:38.797091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.798312 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.798316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:38.799251 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:38.799261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.799983 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.799989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:38.800142 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:38.800894 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:38.800914 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:38.805274 (+ 4360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":258,"threads_started":1}
W20260924 19:13:38.805804 20635 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:38.795458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:40809 (local address 127.11.18.126:45179)
0924 19:13:38.795686 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:38.795695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:38.796060 (+ 365us) server_negotiation.cc:407] Connection header received
0924 19:13:38.796246 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:38.796251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:38.796320 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:38.796446 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:38.797267 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.798161 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.799385 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.799761 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.799859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:38.805554 (+ 5695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:40809: BlockingRecv error: recv got EOF from 127.11.18.65:40809 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 19:13:38 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:38 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:38.837821 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:38.828115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:44515)
0924 19:13:38.828644 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:38.828661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:38.828793 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:38.829232 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:38.829237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:38.829814 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:38.830149 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:38.830158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.831317 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.831321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:38.832256 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:38.832266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.833012 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.833017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:38.833153 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:38.833974 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:38.833991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:38.837541 (+ 3550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":266,"threads_started":1}
W20260924 19:13:38.837932 20635 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:38.828314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:44515 (local address 127.11.18.126:45179)
0924 19:13:38.828547 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:38.828556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:38.828818 (+ 262us) server_negotiation.cc:407] Connection header received
0924 19:13:38.828903 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:38.828909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:38.829007 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:38.829124 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:38.830293 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.831175 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.832438 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:38.832839 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:38.832970 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:38.837740 (+ 4770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:44515: BlockingRecv error: recv got EOF from 127.11.18.67:44515 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 24 19:13:39 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:39 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:39.437517 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:39.428032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:57829)
0924 19:13:39.428589 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:39.428611 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:39.428737 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:39.429166 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:39.429170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:39.429730 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:39.429886 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:39.429894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.430932 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.430936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:39.431893 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:39.431906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.432704 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.432708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:39.432834 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:39.433534 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:39.433552 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:39.437200 (+ 3648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":279,"threads_started":1}
W20260924 19:13:39.437842 20639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:39.428073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:57829 (local address 127.11.18.126:45179)
0924 19:13:39.428811 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:39.428815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:39.428830 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:13:39.428910 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:39.428915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:39.428993 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:39.429097 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:39.430021 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.430747 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.432020 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.432561 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.432653 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:39.437522 (+ 4869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:57829: BlockingRecv error: recv got EOF from 127.11.18.66:57829 (error 108)
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":499,"threads_started":1}
Sep 24 19:13:39 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:39 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:13:39.817130 20639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:39.806775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:36187 (local address 127.11.18.126:45179)
0924 19:13:39.807171 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:39.807177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:39.807434 (+ 257us) server_negotiation.cc:407] Connection header received
0924 19:13:39.807645 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:39.807651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:39.807729 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:39.807873 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:39.808675 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.809599 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.810998 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.811424 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.811571 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:39.816933 (+ 5362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:36187: BlockingRecv error: recv got EOF from 127.11.18.65:36187 (error 108)
Metrics: {"server-negotiator.queue_time_us":224}
I20260924 19:13:39.817126 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:39.806643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:36187)
0924 19:13:39.807322 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:39.807341 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:39.807512 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:39.807905 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:39.807910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:39.808328 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:39.808510 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:39.808519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.809751 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.809757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:39.810838 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:39.810849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.811598 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.811604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:39.811794 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:39.812489 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:39.812514 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:39.816731 (+ 4217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":332,"threads_started":1}
Sep 24 19:13:39 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:39 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:39.850135 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:39.839275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:36997)
0924 19:13:39.839904 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:39.839922 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:39.840112 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:39.840543 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:39.840548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:39.841145 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:39.841311 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:39.841384 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.842798 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.842804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:39.844083 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:39.844096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.844917 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.844924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:39.845117 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:39.845865 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:39.845884 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:39.849906 (+ 4022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":299,"threads_started":1}
W20260924 19:13:39.850577 20639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:39.839355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:36997 (local address 127.11.18.126:45179)
0924 19:13:39.839657 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:39.839666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:39.840031 (+ 365us) server_negotiation.cc:407] Connection header received
0924 19:13:39.840290 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:39.840295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:39.840377 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:39.840498 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:39.841533 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.842633 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.844234 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:39.844749 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:39.844900 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:39.850123 (+ 5223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:36997: BlockingRecv error: recv got EOF from 127.11.18.67:36997 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 24 19:13:40 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:40 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:40.451416 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:40.438383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:41595)
0924 19:13:40.438919 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:40.438936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:40.439041 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:40.439407 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:40.439410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:40.439750 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:40.439956 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:40.439965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.441084 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.441088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:40.442137 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:40.442149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.442940 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.442944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:40.443057 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:40.447408 (+ 4351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:40.447434 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:40.451161 (+ 3727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":254,"threads_started":1}
W20260924 19:13:40.451802 20643 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:40.438492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:41595 (local address 127.11.18.126:45179)
0924 19:13:40.439022 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:40.439027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:40.439044 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:40.439181 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:40.439185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:40.439257 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:40.439349 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:40.440106 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.440926 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.442317 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.442786 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.442914 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:40.451505 (+ 8591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:41595: BlockingRecv error: recv got EOF from 127.11.18.66:41595 (error 108)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":281,"threads_started":1}
Sep 24 19:13:40 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:40 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:40.827620 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:40.818363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:40295)
0924 19:13:40.818916 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:40.818934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:40.819058 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:40.819427 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:40.819431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:40.819830 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:40.820037 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:40.820045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.821102 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.821107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:40.822228 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:40.822240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.823098 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.823106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:40.823286 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:40.823856 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:40.823873 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:40.827368 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
W20260924 19:13:40.827791 20643 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:40.818625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:40295 (local address 127.11.18.126:45179)
0924 19:13:40.818940 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:40.818945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:40.819024 (+ 79us) server_negotiation.cc:407] Connection header received
0924 19:13:40.819170 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:40.819175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:40.819252 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:40.819362 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:40.820183 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.820954 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.822375 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.822936 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.823084 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:40.827572 (+ 4488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:40295: BlockingRecv error: recv got EOF from 127.11.18.65:40295 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 19:13:40 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:40 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:40.861863 20645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:40.851152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:55831)
0924 19:13:40.851691 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:40.851712 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:40.851944 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:40.852306 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:40.852311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:40.852702 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:40.852876 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:40.852885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.854001 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.854007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:40.855143 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:40.855155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.855963 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.855969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:40.856141 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:40.856881 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:40.856900 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:40.861474 (+ 4574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":295,"threads_started":1}
W20260924 19:13:40.861931 20643 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:40.851303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:55831 (local address 127.11.18.126:45179)
0924 19:13:40.851533 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:40.851541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:40.851865 (+ 324us) server_negotiation.cc:407] Connection header received
0924 19:13:40.852078 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:40.852084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:40.852166 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:40.852270 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:40.853018 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.853856 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.855293 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:40.855754 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:40.855933 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:40.861678 (+ 5745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:55831: BlockingRecv error: recv got EOF from 127.11.18.67:55831 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 19:13:41 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:41 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:41.462427 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:41.452427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:43793)
0924 19:13:41.452933 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:41.452950 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:41.453057 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:41.453433 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:41.453436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:41.453929 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:41.454107 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:41.454116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.455292 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.455297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:41.456323 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:41.456404 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.457298 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.457304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:41.457551 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:41.458301 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:41.458321 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:41.462205 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
W20260924 19:13:41.462801 20647 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:41.452580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:43793 (local address 127.11.18.126:45179)
0924 19:13:41.453055 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:41.453060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:41.453077 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:41.453160 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:41.453163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:41.453240 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:41.453402 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:41.454299 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.455127 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.456622 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.457112 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.457263 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:41.462439 (+ 5176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:43793: BlockingRecv error: recv got EOF from 127.11.18.66:43793 (error 108)
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
Sep 24 19:13:41 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:41 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:41.838527 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:41.828755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:42615)
0924 19:13:41.829298 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:41.829315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:41.829452 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:41.829817 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:41.829821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:41.830224 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:41.830390 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:41.830399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.831571 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.831575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:41.832507 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:41.832518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.833313 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.833318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:41.833501 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:41.834262 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:41.834279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:41.838273 (+ 3994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":312,"threads_started":1}
W20260924 19:13:41.838618 20647 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:41.828869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:42615 (local address 127.11.18.126:45179)
0924 19:13:41.829075 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:41.829080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:41.829417 (+ 337us) server_negotiation.cc:407] Connection header received
0924 19:13:41.829555 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:41.829559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:41.829633 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:41.829727 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:41.830534 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.831407 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.832720 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.833145 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.833247 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:41.838448 (+ 5201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:42615: BlockingRecv error: recv got EOF from 127.11.18.65:42615 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 19:13:41 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:41 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:41.872495 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:41.862756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:50373)
0924 19:13:41.863231 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:41.863247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:41.863372 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:41.863773 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:41.863777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:41.864167 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:41.864367 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:41.864377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.865515 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.865519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:41.866485 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:41.866498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.867266 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.867272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:41.867439 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:41.868091 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:41.868111 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:41.872207 (+ 4096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":285,"threads_started":1}
W20260924 19:13:41.872710 20647 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:41.862893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:50373 (local address 127.11.18.126:45179)
0924 19:13:41.863122 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:41.863128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:41.863329 (+ 201us) server_negotiation.cc:407] Connection header received
0924 19:13:41.863464 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:41.863468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:41.863546 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:41.863626 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:41.864520 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.865369 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.866651 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:41.867099 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:41.867202 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:41.872407 (+ 5205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:50373: BlockingRecv error: recv got EOF from 127.11.18.67:50373 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 19:13:42 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:42 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:42.478497 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:42.463477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:34355)
0924 19:13:42.463966 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:42.463984 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:42.464087 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:42.464546 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:42.464550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:42.465009 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:42.465165 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:42.465173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.466308 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.466312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:42.467223 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:42.467234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.468001 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.468005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:42.468136 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:42.468777 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:42.468793 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:42.478159 (+ 9366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":255,"threads_started":1}
W20260924 19:13:42.478971 20651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:42.463554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:34355 (local address 127.11.18.126:45179)
0924 19:13:42.464270 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:42.464274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:42.464289 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:13:42.464332 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:42.464335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:42.464400 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:42.464484 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:42.465308 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.466186 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.467375 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.467803 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.467968 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:42.478403 (+ 10435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:34355: BlockingRecv error: recv got EOF from 127.11.18.66:34355 (error 108)
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":486,"threads_started":1}
Sep 24 19:13:42 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:42 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:42.849637 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:42.839669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:48833)
0924 19:13:42.840270 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:42.840288 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:42.840413 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:42.840808 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:42.840811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:42.841237 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:42.841416 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:42.841425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.842928 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.842935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:42.843968 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:42.843981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.844684 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.844690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:42.844877 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:42.845552 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:42.845569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:42.849258 (+ 3689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":388,"threads_started":1}
W20260924 19:13:42.849701 20651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:42.839785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:48833 (local address 127.11.18.126:45179)
0924 19:13:42.839964 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:42.839968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:42.840399 (+ 431us) server_negotiation.cc:407] Connection header received
0924 19:13:42.840521 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:42.840525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:42.840593 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:42.840774 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:42.841558 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.842787 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.844110 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.844524 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.844667 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:42.849493 (+ 4826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:48833: BlockingRecv error: recv got EOF from 127.11.18.65:48833 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 19:13:42 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:42 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:42.883744 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:42.873806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:45097)
0924 19:13:42.874358 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:42.874377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:42.874496 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:42.874931 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:42.874935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:42.875307 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:42.875489 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:42.875497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.876633 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.876637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:42.877625 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:42.877637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.878339 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.878343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:42.878472 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:42.879222 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:42.879240 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:42.883497 (+ 4257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":308,"threads_started":1}
W20260924 19:13:42.884050 20651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:42.873994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:45097 (local address 127.11.18.126:45179)
0924 19:13:42.874183 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:42.874188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:42.874481 (+ 293us) server_negotiation.cc:407] Connection header received
0924 19:13:42.874634 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:42.874638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:42.874703 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:42.874872 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:42.875651 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.876462 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.877799 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:42.878192 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:42.878302 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:42.883716 (+ 5414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:45097: BlockingRecv error: recv got EOF from 127.11.18.67:45097 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 19:13:43 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:43 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.66@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:43.489118 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:43.479521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.66:45895)
0924 19:13:43.480155 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:43.480179 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:43.480322 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:43.480757 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:43.480762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:43.481166 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:43.481303 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:43.481313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.482473 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.482479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:43.483518 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:43.483530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.484499 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.484505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:43.484709 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:43.485396 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:43.485418 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:43.488886 (+ 3468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":360,"threads_started":1}
W20260924 19:13:43.489454 20655 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:43.479729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:45895 (local address 127.11.18.126:45179)
0924 19:13:43.480439 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:43.480447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:43.480464 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:43.480520 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:43.480523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:43.480582 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:43.480727 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:43.481476 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.482334 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.483701 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.484338 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.484464 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:43.489096 (+ 4632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.66:45895: BlockingRecv error: recv got EOF from 127.11.18.66:45895 (error 108)
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":463,"threads_started":1}
Sep 24 19:13:43 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:43 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.65@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:43.861209 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:43.850902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.65:57879)
0924 19:13:43.851590 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:43.851606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:43.851719 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:43.852045 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:43.852048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:43.852450 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:43.852614 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:43.852622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.853682 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.853688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:43.854745 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:43.854759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.855473 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.855478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:43.855608 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:43.856367 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:43.856386 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:43.860959 (+ 4573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":410,"threads_started":1}
W20260924 19:13:43.861461 20655 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:43.850961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:57879 (local address 127.11.18.126:45179)
0924 19:13:43.851148 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:43.851153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:43.851684 (+ 531us) server_negotiation.cc:407] Connection header received
0924 19:13:43.851855 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:43.851859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:43.851923 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:43.852008 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:43.852743 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.853511 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.854921 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.855330 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.855427 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:43.861202 (+ 5775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.65:57879: BlockingRecv error: recv got EOF from 127.11.18.65:57879 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 19:13:43 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:13:43 dist-test-slave-0lgr krb5kdc[19596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.18.67@KRBTEST.COM for oryx/127.11.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:13:43.894047 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:43.884758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45179 (local address 127.11.18.67:52687)
0924 19:13:43.885281 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:43.885303 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:43.885471 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:43.885770 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:43.885773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:43.886160 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:43.886336 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:43.886347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.887809 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.887814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:43.888945 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:43.888958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.889684 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.889688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:43.889806 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:43.890449 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:43.890466 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:43.893825 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.18.126:45179: Server oryx/127.11.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":296,"threads_started":1}
W20260924 19:13:43.894308 20655 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:43.884921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:52687 (local address 127.11.18.126:45179)
0924 19:13:43.885181 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:43.885186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:43.885413 (+ 227us) server_negotiation.cc:407] Connection header received
0924 19:13:43.885578 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:43.885582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:43.885644 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:43.885791 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:43.886501 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.887633 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.889072 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:43.889561 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:43.889656 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:43.894027 (+ 4371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.18.67:52687: BlockingRecv error: recv got EOF from 127.11.18.67:52687 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
I20260924 19:13:44.374552 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20185
I20260924 19:13:44.388413 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:44.376010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.11.18.65:40955)
0924 19:13:44.376660 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:44.376667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:44.376693 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:13:44.376758 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:44.376764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:44.376992 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:44.377186 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:44.378877 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:44.380047 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:44.381501 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:44.382095 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:44.382210 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:44.384883 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:44.384892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:44.384907 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:44.384998 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:44.387891 (+ 2893us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.18.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:13:44.388204 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38532: Request ticket server kudu/127.11.18.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":388,"threads_started":1}
W20260924 19:13:44.388506 20659 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38532: Request ticket server kudu/127.11.18.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 19:13:44.388818 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20185: Not authorized: Client connection negotiation failed: client connection to 127.11.18.65:40955: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.18.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:13:44.389075 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 20185
I20260924 19:13:44.406858 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20336
I20260924 19:13:44.420096 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:44.408098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.11.18.66:46353)
0924 19:13:44.408706 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:44.408714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:44.408740 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:13:44.408802 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:44.408808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:44.409073 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:44.409310 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:44.410739 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:44.411744 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:44.412976 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:44.413608 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:44.413758 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:44.416466 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:44.416476 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:44.416491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:44.416540 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:44.419357 (+ 2817us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.18.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:13:44.419735 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33248: Request ticket server kudu/127.11.18.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":319,"threads_started":1}
W20260924 19:13:44.420151 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20336: Not authorized: Client connection negotiation failed: client connection to 127.11.18.66:46353: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.18.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 19:13:44.420212 20660 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33248: Request ticket server kudu/127.11.18.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:13:44.420297 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 20336
I20260924 19:13:44.434978 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20469
I20260924 19:13:44.449255 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:44.436170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.11.18.67:42487)
0924 19:13:44.436704 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:44.436712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:44.436738 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:13:44.436806 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:44.436812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:44.437085 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:44.437253 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:44.439420 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:44.440530 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:44.442019 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:44.442634 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:44.442765 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:44.445652 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:44.445662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:44.445678 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:44.445739 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:44.448712 (+ 2973us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.18.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:13:44.449006 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60554: Request ticket server kudu/127.11.18.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
W20260924 19:13:44.449640 20661 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60554: Request ticket server kudu/127.11.18.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 19:13:44.449666 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20469: Not authorized: Client connection negotiation failed: client connection to 127.11.18.67:42487: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.18.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:13:44.449833 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 20469
I20260924 19:13:44.466288 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20116
I20260924 19:13:44.467643 20178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:44.585186 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 20116
2026-09-24T19:13:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21480 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T19:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:13:44Z Disabled control of system clock
I20260924 19:13:44.665171 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:46755
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:46755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:13:45.002673 20666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:45.002990 20666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:45.003121 20666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:45.010582 20666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:45.010700 20666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:45.010772 20666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:45.010888 20666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:45.010962 20666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:45.020313 20666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:46755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:46755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:45.021668 20666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:45.023435 20666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:45.035610 20674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:45.035717 20666 server_base.cc:1034] running on GCE node
W20260924 19:13:45.035861 20671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:45.035653 20672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:45.036861 20666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:45.038375 20666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:45.039633 20666 hybrid_clock.cc:648] HybridClock initialized: now 1790277225039591 us; error 37 us; skew 500 ppm
I20260924 19:13:45.040095 20666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:13:45.042608 20666 webserver.cc:460] Webserver started at http://127.11.18.126:33765/ using document root <none> and password file <none>
I20260924 19:13:45.043352 20666 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:45.043502 20666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:45.043852 20666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:45.046880 20666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986"
format_stamp: "Formatted at 2026-09-24 19:13:45 on dist-test-slave-0lgr"
server_key: "a96657f230576b4012cfe546a2b0782f"
server_key_iv: "5930f178794bbd7706643917a0e78d6c"
server_key_version: "encryptionkey@0"
I20260924 19:13:45.047720 20666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986"
format_stamp: "Formatted at 2026-09-24 19:13:45 on dist-test-slave-0lgr"
server_key: "a96657f230576b4012cfe546a2b0782f"
server_key_iv: "5930f178794bbd7706643917a0e78d6c"
server_key_version: "encryptionkey@0"
I20260924 19:13:45.053812 20666 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 19:13:45.057789 20680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:45.059512 20666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:13:45.059748 20666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986"
format_stamp: "Formatted at 2026-09-24 19:13:45 on dist-test-slave-0lgr"
server_key: "a96657f230576b4012cfe546a2b0782f"
server_key_iv: "5930f178794bbd7706643917a0e78d6c"
server_key_version: "encryptionkey@0"
I20260924 19:13:45.060026 20666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:45.092497 20666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:45.093407 20666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:45.093703 20666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:45.112777 20666 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:46755
I20260924 19:13:45.112802 20731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:46755 every 8 connection(s)
I20260924 19:13:45.114240 20666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:13:45.118234 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:45.120078 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 20666
I20260924 19:13:45.120275 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:13:45.121110 11337 external_mini_cluster.cc:1428] Setting key 834c7dd81a7d416a38e5cf6c889a5205
I20260924 19:13:45.130826 20732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986: Bootstrap starting.
I20260924 19:13:45.134614 20732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:45.135738 20732 log.cc:824] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:45.135870 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:45.123323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.11.18.126:46755)
0924 19:13:45.124398 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:45.124410 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:45.124452 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:13:45.126157 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:45.126182 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:45.126673 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:45.127028 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:45.127706 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:45.129081 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:45.131061 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:45.131836 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:45.131983 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:45.133501 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:45.133508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:45.133523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:45.134084 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:45.134174 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:45.134197 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:45.134331 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:45.134532 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 19:13:45.134874 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":386,"threads_started":1}
I20260924 19:13:45.138806 20732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986: No bootstrap required, opened a new log
I20260924 19:13:45.146099 20732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46755 } }
I20260924 19:13:45.146535 20732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:45.146658 20732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b63a6ab9b8b4014b1bceebc5dbb3986, State: Initialized, Role: FOLLOWER
I20260924 19:13:45.147320 20732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46755 } }
I20260924 19:13:45.147569 20732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:13:45.147727 20732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:13:45.147907 20732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:45.150501 20732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46755 } }
I20260924 19:13:45.151042 20732 leader_election.cc:302] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b63a6ab9b8b4014b1bceebc5dbb3986; no voters:
I20260924 19:13:45.151676 20732 leader_election.cc:288] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:13:45.151849 20737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:45.152935 20737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [term 1 LEADER]: Becoming Leader. State: Replica: 5b63a6ab9b8b4014b1bceebc5dbb3986, State: Running, Role: LEADER
I20260924 19:13:45.153589 20737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46755 } }
I20260924 19:13:45.154093 20732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:45.156445 20738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46755 } } }
I20260924 19:13:45.156554 20739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b63a6ab9b8b4014b1bceebc5dbb3986. Latest consensus state: current_term: 1 leader_uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b63a6ab9b8b4014b1bceebc5dbb3986" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46755 } } }
I20260924 19:13:45.156953 20739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:45.156836 20738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:45.158582 20745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:13:45.164160 20745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:13:45.175532 20745 catalog_manager.cc:1347] Generated new cluster ID: 7d951a24b5e8421aa029ea113d9d50f8
I20260924 19:13:45.175661 20745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:13:45.202035 20745 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:13:45.203606 20745 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:13:45.215183 20745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b63a6ab9b8b4014b1bceebc5dbb3986: Generated new TSK 0
I20260924 19:13:45.216152 20745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:13:45.222615 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46755
--builtin_ntp_servers=127.11.18.84:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:13:45.588855 20756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:45.589150 20756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:45.589234 20756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:45.596766 20756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:45.596886 20756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:45.597065 20756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:45.606510 20756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:45.607767 20756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:45.609527 20756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:45.624229 20764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:45.624163 20762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:45.624203 20761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:45.626089 20756 server_base.cc:1034] running on GCE node
I20260924 19:13:45.626870 20756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:45.628026 20756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:45.629287 20756 hybrid_clock.cc:648] HybridClock initialized: now 1790277225629242 us; error 70 us; skew 500 ppm
I20260924 19:13:45.629738 20756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:13:45.638113 20756 webserver.cc:460] Webserver started at http://127.11.18.65:39109/ using document root <none> and password file <none>
I20260924 19:13:45.639047 20756 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:45.639195 20756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:45.639564 20756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:45.642885 20756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "474b3fce06bc4d5f96d9cea52452073b"
format_stamp: "Formatted at 2026-09-24 19:13:45 on dist-test-slave-0lgr"
server_key: "16adafa14f808e89a2493bb168e021cf"
server_key_iv: "a67041a083f2289ecd6874b3ee6edabf"
server_key_version: "encryptionkey@0"
I20260924 19:13:45.643801 20756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "474b3fce06bc4d5f96d9cea52452073b"
format_stamp: "Formatted at 2026-09-24 19:13:45 on dist-test-slave-0lgr"
server_key: "16adafa14f808e89a2493bb168e021cf"
server_key_iv: "a67041a083f2289ecd6874b3ee6edabf"
server_key_version: "encryptionkey@0"
I20260924 19:13:45.650032 20756 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260924 19:13:45.654073 20770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:45.655793 20756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:45.655992 20756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "474b3fce06bc4d5f96d9cea52452073b"
format_stamp: "Formatted at 2026-09-24 19:13:45 on dist-test-slave-0lgr"
server_key: "16adafa14f808e89a2493bb168e021cf"
server_key_iv: "a67041a083f2289ecd6874b3ee6edabf"
server_key_version: "encryptionkey@0"
I20260924 19:13:45.656226 20756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:45.676218 20756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:45.677278 20756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:45.677654 20756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:45.678733 20756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:45.680207 20756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:45.680328 20756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:45.680456 20756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:45.680540 20756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:45.717303 20756 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:42851
I20260924 19:13:45.717463 20882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:42851 every 8 connection(s)
I20260924 19:13:45.718794 20756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:45.722596 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 20756
I20260924 19:13:45.722812 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:13:45.723703 11337 external_mini_cluster.cc:1428] Setting key 3c87858b65aaa4a38863119b42ca0be5
I20260924 19:13:45.726729 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46755
--builtin_ntp_servers=127.11.18.84:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:13:45.734799 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:45.721911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:35019 (local address 127.11.18.126:46755)
0924 19:13:45.722358 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:45.722365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:45.723927 (+ 1562us) server_negotiation.cc:407] Connection header received
0924 19:13:45.726092 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:45.726096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:45.726157 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:45.726260 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:45.727746 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:45.728560 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:45.731080 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:45.731794 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:45.731948 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:45.733716 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:45.733727 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:45.733736 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:45.733768 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:45.733825 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:45.733847 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:45.733955 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:45.734198 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 19:13:45.734319 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":226,"threads_started":1}
I20260924 19:13:45.735826 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:45.722541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46755 (local address 127.11.18.65:35019)
0924 19:13:45.723706 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:45.723780 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:45.725792 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:45.726542 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:45.726557 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:45.726681 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:45.727541 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:45.727571 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:45.728731 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:45.728747 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:45.730894 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:45.730911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:45.732051 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:45.732060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:45.732393 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:45.733252 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:13:45.733256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:45.733321 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:13:45.733475 (+ 154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:45.733959 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:45.733967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:45.733974 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:13:45.734121 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 19:13:45.734438 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":370,"threads_started":1}
I20260924 19:13:45.737560 20883 heartbeater.cc:347] Connected to a master server at 127.11.18.126:46755
I20260924 19:13:45.737900 20883 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:45.738830 20883 heartbeater.cc:511] Master 127.11.18.126:46755 requested a full tablet report, sending...
I20260924 19:13:45.741392 20697 ts_manager.cc:194] Registered new tserver with Master: 474b3fce06bc4d5f96d9cea52452073b (127.11.18.65:42851)
I20260924 19:13:45.743633 20697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.65:35019
W20260924 19:13:46.092859 20887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:46.093171 20887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:46.093253 20887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:46.100513 20887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:46.100629 20887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:46.100735 20887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:46.109880 20887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:46.111153 20887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:46.112942 20887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:46.127002 20892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:46.126937 20893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:46.130658 20895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:46.131189 20887 server_base.cc:1034] running on GCE node
I20260924 19:13:46.131989 20887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:46.133090 20887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:46.134325 20887 hybrid_clock.cc:648] HybridClock initialized: now 1790277226134271 us; error 85 us; skew 500 ppm
I20260924 19:13:46.134630 20887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:13:46.137060 20887 webserver.cc:460] Webserver started at http://127.11.18.66:35771/ using document root <none> and password file <none>
I20260924 19:13:46.137842 20887 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:46.137933 20887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:46.138191 20887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:46.141055 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "abb10d69cfe44222b4f6c0b6bb2e1a3c"
format_stamp: "Formatted at 2026-09-24 19:13:46 on dist-test-slave-0lgr"
server_key: "18b7ca1392d968eb47d79bea8f7df591"
server_key_iv: "dab4f92665c9804043ad32bdbced2065"
server_key_version: "encryptionkey@0"
I20260924 19:13:46.141914 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "abb10d69cfe44222b4f6c0b6bb2e1a3c"
format_stamp: "Formatted at 2026-09-24 19:13:46 on dist-test-slave-0lgr"
server_key: "18b7ca1392d968eb47d79bea8f7df591"
server_key_iv: "dab4f92665c9804043ad32bdbced2065"
server_key_version: "encryptionkey@0"
I20260924 19:13:46.147636 20887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:13:46.152122 20901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:46.154076 20887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:13:46.154286 20887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "abb10d69cfe44222b4f6c0b6bb2e1a3c"
format_stamp: "Formatted at 2026-09-24 19:13:46 on dist-test-slave-0lgr"
server_key: "18b7ca1392d968eb47d79bea8f7df591"
server_key_iv: "dab4f92665c9804043ad32bdbced2065"
server_key_version: "encryptionkey@0"
I20260924 19:13:46.154515 20887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:46.170804 20887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:46.171660 20887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:46.171965 20887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:46.173012 20887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:46.174523 20887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:46.174649 20887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:46.174798 20887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:46.174886 20887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:46.209168 20887 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:44199
I20260924 19:13:46.209259 21013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:44199 every 8 connection(s)
I20260924 19:13:46.210512 20887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:46.215938 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 20887
I20260924 19:13:46.216111 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:13:46.216588 11337 external_mini_cluster.cc:1428] Setting key 329de039b8f342c16dfdb1c0a557dfbb
I20260924 19:13:46.220028 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46755
--builtin_ntp_servers=127.11.18.84:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:13:46.225147 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:46.213416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:44241 (local address 127.11.18.126:46755)
0924 19:13:46.213635 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:46.213640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:46.215187 (+ 1547us) server_negotiation.cc:407] Connection header received
0924 19:13:46.217228 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:46.217234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:46.217341 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:46.217479 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:46.218949 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.220065 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.222002 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.222710 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.222851 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:46.224206 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:46.224215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:46.224223 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:46.224250 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:46.224291 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:46.224308 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:46.224406 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:46.224632 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:13:46.224743 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:13:46.226078 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:46.214044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46755 (local address 127.11.18.66:44241)
0924 19:13:46.215011 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:46.215053 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:46.216925 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:46.217845 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:46.217857 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:46.217973 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:46.218713 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:46.218740 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.220285 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.220294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:46.221853 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:46.221866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.222965 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.222970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:46.223176 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:46.223883 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:13:46.223886 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:46.223926 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:13:46.224006 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:46.224415 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:46.224425 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:46.224431 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:13:46.224588 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 19:13:46.224961 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":340,"threads_started":1}
I20260924 19:13:46.227928 21014 heartbeater.cc:347] Connected to a master server at 127.11.18.126:46755
I20260924 19:13:46.228288 21014 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:46.229419 21014 heartbeater.cc:511] Master 127.11.18.126:46755 requested a full tablet report, sending...
I20260924 19:13:46.231243 20697 ts_manager.cc:194] Registered new tserver with Master: abb10d69cfe44222b4f6c0b6bb2e1a3c (127.11.18.66:44199)
I20260924 19:13:46.232106 20697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.66:44241
W20260924 19:13:46.556257 21017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:46.556618 21017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:46.556782 21017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:46.564769 21017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:46.564908 21017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:46.565092 21017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:46.574232 21017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:46.575559 21017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:46.577432 21017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:46.591550 21022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:46.591619 21023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:46.591619 21025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:46.593052 21017 server_base.cc:1034] running on GCE node
I20260924 19:13:46.594327 21017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:46.595710 21017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:46.596956 21017 hybrid_clock.cc:648] HybridClock initialized: now 1790277226596896 us; error 53 us; skew 500 ppm
I20260924 19:13:46.597467 21017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:13:46.600178 21017 webserver.cc:460] Webserver started at http://127.11.18.67:38419/ using document root <none> and password file <none>
I20260924 19:13:46.600934 21017 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:46.601051 21017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:46.601455 21017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:46.604362 21017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "82eb054697fa4e12987db3e62b00dd07"
format_stamp: "Formatted at 2026-09-24 19:13:46 on dist-test-slave-0lgr"
server_key: "a9fd6c26cc300efac83146c96bbca267"
server_key_iv: "d31dd49355c1499c4c15128ad1af8188"
server_key_version: "encryptionkey@0"
I20260924 19:13:46.605207 21017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "82eb054697fa4e12987db3e62b00dd07"
format_stamp: "Formatted at 2026-09-24 19:13:46 on dist-test-slave-0lgr"
server_key: "a9fd6c26cc300efac83146c96bbca267"
server_key_iv: "d31dd49355c1499c4c15128ad1af8188"
server_key_version: "encryptionkey@0"
I20260924 19:13:46.610570 21017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:13:46.614188 21031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:46.615738 21017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:46.615922 21017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "82eb054697fa4e12987db3e62b00dd07"
format_stamp: "Formatted at 2026-09-24 19:13:46 on dist-test-slave-0lgr"
server_key: "a9fd6c26cc300efac83146c96bbca267"
server_key_iv: "d31dd49355c1499c4c15128ad1af8188"
server_key_version: "encryptionkey@0"
I20260924 19:13:46.616139 21017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:46.630434 21017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:46.631304 21017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:46.631560 21017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:46.632498 21017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:46.633988 21017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:46.634073 21017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:46.634147 21017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:46.634202 21017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:46.670929 21017 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:45857
I20260924 19:13:46.671020 21143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:45857 every 8 connection(s)
I20260924 19:13:46.672225 21017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:46.677450 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 21017
I20260924 19:13:46.677619 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:13:46.678047 11337 external_mini_cluster.cc:1428] Setting key 83d7460ce61a24d0e21b6ce34196884d
I20260924 19:13:46.684732 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:46.674797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:33743 (local address 127.11.18.126:46755)
0924 19:13:46.674986 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:46.674991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:46.676366 (+ 1375us) server_negotiation.cc:407] Connection header received
0924 19:13:46.678279 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:46.678284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:46.678376 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:46.678489 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:46.679669 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.680470 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.682059 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.682463 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.682593 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:46.683909 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:46.683915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:46.683918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:46.683956 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:46.683994 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:46.684010 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:46.684112 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:46.684328 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 19:13:46.684419 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:13:46.685784 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:46.675260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46755 (local address 127.11.18.67:33743)
0924 19:13:46.676167 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:46.676220 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:46.677976 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:46.678717 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:46.678727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:46.678840 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:46.679516 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:46.679533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.680618 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.680623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:46.681921 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:46.681932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.682672 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.682680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:46.682899 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:46.683528 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:13:46.683530 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:46.683567 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:13:46.683640 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:46.684110 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:46.684121 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:46.684130 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 19:13:46.684356 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 19:13:46.684720 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":286,"threads_started":1}
I20260924 19:13:46.687129 21144 heartbeater.cc:347] Connected to a master server at 127.11.18.126:46755
I20260924 19:13:46.687418 21144 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:46.688073 21144 heartbeater.cc:511] Master 127.11.18.126:46755 requested a full tablet report, sending...
I20260924 19:13:46.689438 20697 ts_manager.cc:194] Registered new tserver with Master: 82eb054697fa4e12987db3e62b00dd07 (127.11.18.67:45857)
I20260924 19:13:46.690186 20697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.67:33743
I20260924 19:13:46.693126 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:13:46.700302 20885 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:46.698386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.11.18.126:46755)
0924 19:13:46.698587 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:46.698596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:46.699107 (+ 511us) server_negotiation.cc:407] Connection header received
0924 19:13:46.699355 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:46.699361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:46.699472 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:46.699590 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:46.700044 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36208: BlockingRecv error: recv got EOF from 127.0.0.1:36208 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:13:46.701929 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20756
I20260924 19:13:46.710414 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:46.702902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.11.18.65:42851)
0924 19:13:46.703437 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:46.703454 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:46.703490 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:13:46.703553 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:46.703560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:46.703776 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:46.703913 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:46.704354 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.705677 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.707146 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.707753 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.707864 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:46.708952 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:46.708960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:46.708973 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:46.709699 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:46.709748 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:46.709774 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:46.709861 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:46.710000 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:13:46.710123 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":278,"threads_started":1}
I20260924 19:13:46.711790 20878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:46.794647 20883 heartbeater.cc:503] Master 127.11.18.126:46755 was elected leader, sending a full tablet report...
I20260924 19:13:46.795624 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 20756
W20260924 19:13:46.812431 20681 connection.cc:535] server connection from 127.11.18.65:35019 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:13:46.812834 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20887
I20260924 19:13:46.821650 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:46.813861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.11.18.66:44199)
0924 19:13:46.814459 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:46.814467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:46.814495 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:13:46.814554 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:46.814560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:46.814863 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:46.814992 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:46.815350 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.816417 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.817805 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.818448 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.818561 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:46.819740 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:46.819750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:46.819765 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:46.820644 (+ 879us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:46.820695 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:46.820717 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:46.820829 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:46.821037 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 19:13:46.821254 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":299,"threads_started":1}
I20260924 19:13:46.823100 21009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:46.908737 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 20887
I20260924 19:13:46.924834 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 21017
I20260924 19:13:46.933728 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:46.926149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.11.18.67:45857)
0924 19:13:46.926830 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:46.926837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:46.926867 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:13:46.926927 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:46.926933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:46.927143 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:46.927308 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:46.927730 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.928781 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.930220 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:46.930965 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:46.931070 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:46.932294 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:46.932303 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:46.932321 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:46.932907 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:46.932952 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:46.932977 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:46.933072 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:46.933220 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 19:13:46.933421 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":359,"threads_started":1}
I20260924 19:13:46.935153 21139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:47.019132 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 21017
I20260924 19:13:47.036886 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 20666
I20260924 19:13:47.038300 20727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:47.142056 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 20666
2026-09-24T19:13:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2552 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T19:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:13:47Z Disabled control of system clock
I20260924 19:13:47.216499 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:40233
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:13:47.564739 21163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:47.565065 21163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:47.565152 21163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:47.572579 21163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:47.572805 21163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:47.572850 21163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:47.572901 21163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:47.572942 21163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:47.582578 21163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:40233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:47.583896 21163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:47.585704 21163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:47.598140 21168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:47.598115 21169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:47.598353 21171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:47.598903 21163 server_base.cc:1034] running on GCE node
I20260924 19:13:47.600059 21163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:47.601590 21163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:47.602823 21163 hybrid_clock.cc:648] HybridClock initialized: now 1790277227602759 us; error 56 us; skew 500 ppm
I20260924 19:13:47.603186 21163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:13:47.605727 21163 webserver.cc:460] Webserver started at http://127.11.18.126:44241/ using document root <none> and password file <none>
I20260924 19:13:47.606438 21163 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:47.606552 21163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:47.606875 21163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:47.609925 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "372d4edc84694fb0bb0e4d1c38cd95c1"
format_stamp: "Formatted at 2026-09-24 19:13:47 on dist-test-slave-0lgr"
server_key: "6134345965d1703583e01df1f1381c48"
server_key_iv: "ace1e5013de3812fc3e4ae8031056aa5"
server_key_version: "encryptionkey@0"
I20260924 19:13:47.610765 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "372d4edc84694fb0bb0e4d1c38cd95c1"
format_stamp: "Formatted at 2026-09-24 19:13:47 on dist-test-slave-0lgr"
server_key: "6134345965d1703583e01df1f1381c48"
server_key_iv: "ace1e5013de3812fc3e4ae8031056aa5"
server_key_version: "encryptionkey@0"
I20260924 19:13:47.616058 21163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 19:13:47.619837 21177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:47.621493 21163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:13:47.621693 21163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "372d4edc84694fb0bb0e4d1c38cd95c1"
format_stamp: "Formatted at 2026-09-24 19:13:47 on dist-test-slave-0lgr"
server_key: "6134345965d1703583e01df1f1381c48"
server_key_iv: "ace1e5013de3812fc3e4ae8031056aa5"
server_key_version: "encryptionkey@0"
I20260924 19:13:47.621932 21163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:47.627883 21163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:47.628695 21163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:47.629024 21163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:47.648547 21163 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:40233
I20260924 19:13:47.648588 21228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:40233 every 8 connection(s)
I20260924 19:13:47.650012 21163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:13:47.653927 21229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:47.660072 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 21163
I20260924 19:13:47.660303 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:13:47.660975 11337 external_mini_cluster.cc:1428] Setting key 4b1e1e734ffb5a1fa9ca37dbdb123662
I20260924 19:13:47.665092 21229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1: Bootstrap starting.
I20260924 19:13:47.668814 21229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:47.669965 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:47.663330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.11.18.126:40233)
0924 19:13:47.664348 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:47.664359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:47.664404 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:13:47.666188 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:47.666214 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:47.666663 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:47.667015 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:47.668196 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:47.668203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:47.668216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:47.668237 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:47.668315 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:47.668332 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:47.668479 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:47.668640 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:13:47.668921 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":348,"threads_started":1}
I20260924 19:13:47.670415 21229 log.cc:824] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:47.673772 21229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1: No bootstrap required, opened a new log
I20260924 19:13:47.681397 21229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40233 } }
I20260924 19:13:47.681790 21229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:47.681893 21229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372d4edc84694fb0bb0e4d1c38cd95c1, State: Initialized, Role: FOLLOWER
I20260924 19:13:47.683041 21229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40233 } }
I20260924 19:13:47.683475 21234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:47.683756 21234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40233 } }
I20260924 19:13:47.684468 21234 leader_election.cc:302] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372d4edc84694fb0bb0e4d1c38cd95c1; no voters:
I20260924 19:13:47.685307 21234 leader_election.cc:288] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260924 19:13:47.683373 21229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:13:47.685592 21235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:47.685765 21229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:13:47.685873 21229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:47.688347 21229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40233 } }
I20260924 19:13:47.688628 21229 leader_election.cc:302] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372d4edc84694fb0bb0e4d1c38cd95c1; no voters:
I20260924 19:13:47.688649 21235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:47.688853 21235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:13:47.688951 21229 leader_election.cc:288] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:13:47.691540 21229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:47.692572 21235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40233 } }
I20260924 19:13:47.693452 21234 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won
I20260924 19:13:47.694037 21235 leader_election.cc:302] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372d4edc84694fb0bb0e4d1c38cd95c1; no voters:
I20260924 19:13:47.694289 21235 leader_election.cc:288] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:13:47.694370 21234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:13:47.694622 21234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [term 2 LEADER]: Becoming Leader. State: Replica: 372d4edc84694fb0bb0e4d1c38cd95c1, State: Running, Role: LEADER
I20260924 19:13:47.694687 21236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40233 } } }
I20260924 19:13:47.695118 21236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:13:47.695374 21234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40233 } }
I20260924 19:13:47.697980 21235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372d4edc84694fb0bb0e4d1c38cd95c1. Latest consensus state: current_term: 2 leader_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d4edc84694fb0bb0e4d1c38cd95c1" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40233 } } }
I20260924 19:13:47.698243 21235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:47.699234 21243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:13:47.710415 21243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:13:47.720719 21243 catalog_manager.cc:1347] Generated new cluster ID: be514f4b1794442aa6fdf1949234f7bc
I20260924 19:13:47.720881 21243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:13:47.746881 21243 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:13:47.747406 21243 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:13:47.757035 21243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 372d4edc84694fb0bb0e4d1c38cd95c1: Generated new TSK 0
I20260924 19:13:47.758253 21243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:13:47.771986 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40233
--builtin_ntp_servers=127.11.18.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:13:48.108747 21253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:48.109073 21253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:48.109169 21253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:48.116770 21253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:48.116897 21253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:48.117012 21253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:48.126917 21253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:48.128207 21253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:48.130023 21253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:48.144224 21258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:48.144279 21259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:48.144196 21261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:48.145443 21253 server_base.cc:1034] running on GCE node
I20260924 19:13:48.146095 21253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:48.147106 21253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:48.148367 21253 hybrid_clock.cc:648] HybridClock initialized: now 1790277228148336 us; error 57 us; skew 500 ppm
I20260924 19:13:48.148679 21253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:13:48.151351 21253 webserver.cc:460] Webserver started at http://127.11.18.65:38891/ using document root <none> and password file <none>
I20260924 19:13:48.152087 21253 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:48.152179 21253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:48.152438 21253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:48.157125 21253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "b799d4c8353e4101a065a49b77d94ff2"
format_stamp: "Formatted at 2026-09-24 19:13:48 on dist-test-slave-0lgr"
server_key: "6ad86b0ed9fd3a84cb85333f38c6eff9"
server_key_iv: "be409dfdcf83a12e18c14995023851e9"
server_key_version: "encryptionkey@0"
I20260924 19:13:48.158054 21253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "b799d4c8353e4101a065a49b77d94ff2"
format_stamp: "Formatted at 2026-09-24 19:13:48 on dist-test-slave-0lgr"
server_key: "6ad86b0ed9fd3a84cb85333f38c6eff9"
server_key_iv: "be409dfdcf83a12e18c14995023851e9"
server_key_version: "encryptionkey@0"
I20260924 19:13:48.163357 21253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:13:48.167342 21267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:48.168952 21253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:13:48.169288 21253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "b799d4c8353e4101a065a49b77d94ff2"
format_stamp: "Formatted at 2026-09-24 19:13:48 on dist-test-slave-0lgr"
server_key: "6ad86b0ed9fd3a84cb85333f38c6eff9"
server_key_iv: "be409dfdcf83a12e18c14995023851e9"
server_key_version: "encryptionkey@0"
I20260924 19:13:48.169608 21253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:48.176045 21253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:48.176971 21253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:48.177273 21253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:48.178329 21253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:48.179780 21253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:48.179869 21253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:48.179955 21253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:48.180015 21253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:48.219939 21253 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:37297
I20260924 19:13:48.219990 21379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:37297 every 8 connection(s)
I20260924 19:13:48.221437 21253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:48.226611 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 21253
I20260924 19:13:48.226850 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:13:48.227478 11337 external_mini_cluster.cc:1428] Setting key 40f24124f3d710aee1af191512ecc5d3
I20260924 19:13:48.230346 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:48.224115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:44437 (local address 127.11.18.126:40233)
0924 19:13:48.224619 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:48.224627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:48.225755 (+ 1128us) server_negotiation.cc:407] Connection header received
0924 19:13:48.227748 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:48.227754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:48.227824 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:48.227952 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:48.229543 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:48.229549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:48.229553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:48.229572 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:48.229621 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:48.229633 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:48.229754 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:48.229965 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 19:13:48.230030 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
I20260924 19:13:48.230448 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40233
--builtin_ntp_servers=127.11.18.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 19:13:48.231252 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:48.224577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:40233 (local address 127.11.18.65:44437)
0924 19:13:48.225598 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:48.225641 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:48.227436 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:48.228199 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:48.228207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:48.228351 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:48.229135 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:13:48.229141 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:48.229196 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:13:48.229317 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:48.229726 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:48.229737 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:48.229744 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:13:48.230035 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 19:13:48.230288 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":280,"threads_started":1}
I20260924 19:13:48.232909 21380 heartbeater.cc:347] Connected to a master server at 127.11.18.126:40233
I20260924 19:13:48.233220 21380 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:48.233830 21380 heartbeater.cc:511] Master 127.11.18.126:40233 requested a full tablet report, sending...
I20260924 19:13:48.236099 21194 ts_manager.cc:194] Registered new tserver with Master: b799d4c8353e4101a065a49b77d94ff2 (127.11.18.65:37297)
W20260924 19:13:48.567479 21384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:48.567788 21384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:48.567879 21384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:48.576043 21384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:48.576175 21384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:48.576351 21384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:48.585990 21384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:48.587332 21384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:48.589118 21384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:48.603165 21390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:48.603205 21392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:48.603685 21389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:48.604193 21384 server_base.cc:1034] running on GCE node
I20260924 19:13:48.604787 21384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:48.605896 21384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:48.607105 21384 hybrid_clock.cc:648] HybridClock initialized: now 1790277228607034 us; error 63 us; skew 500 ppm
I20260924 19:13:48.607447 21384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:13:48.610014 21384 webserver.cc:460] Webserver started at http://127.11.18.66:36925/ using document root <none> and password file <none>
I20260924 19:13:48.610792 21384 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:48.610885 21384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:48.611235 21384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:48.614204 21384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "f1f4f114ff3d41759a532e40ee03b6be"
format_stamp: "Formatted at 2026-09-24 19:13:48 on dist-test-slave-0lgr"
server_key: "54f625e9780c36e955d7db9b25a76cba"
server_key_iv: "f39aa9c43dd6c07926d8469c21ca4ade"
server_key_version: "encryptionkey@0"
I20260924 19:13:48.615001 21384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "f1f4f114ff3d41759a532e40ee03b6be"
format_stamp: "Formatted at 2026-09-24 19:13:48 on dist-test-slave-0lgr"
server_key: "54f625e9780c36e955d7db9b25a76cba"
server_key_iv: "f39aa9c43dd6c07926d8469c21ca4ade"
server_key_version: "encryptionkey@0"
I20260924 19:13:48.620487 21384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 19:13:48.624192 21398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:48.625836 21384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:13:48.626034 21384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "f1f4f114ff3d41759a532e40ee03b6be"
format_stamp: "Formatted at 2026-09-24 19:13:48 on dist-test-slave-0lgr"
server_key: "54f625e9780c36e955d7db9b25a76cba"
server_key_iv: "f39aa9c43dd6c07926d8469c21ca4ade"
server_key_version: "encryptionkey@0"
I20260924 19:13:48.626296 21384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:48.631997 21384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:48.632748 21384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:48.633018 21384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:48.634114 21384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:48.635521 21384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:48.635635 21384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:48.635764 21384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:48.635843 21384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:48.677162 21384 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:34217
I20260924 19:13:48.677237 21510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:34217 every 8 connection(s)
I20260924 19:13:48.678527 21384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:48.685057 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 21384
I20260924 19:13:48.685263 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:13:48.685844 11337 external_mini_cluster.cc:1428] Setting key 7edc0fc352261cc37ffdf1b10f8d4690
I20260924 19:13:48.687592 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:48.681133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:46359 (local address 127.11.18.126:40233)
0924 19:13:48.681480 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:48.681486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:48.682939 (+ 1453us) server_negotiation.cc:407] Connection header received
0924 19:13:48.684694 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:48.684698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:48.684752 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:48.684845 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:48.686656 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:48.686663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:48.686668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:48.686802 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:48.686847 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:48.686868 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:48.687002 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:48.687206 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 19:13:48.687270 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260924 19:13:48.688516 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:48.681825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:40233 (local address 127.11.18.66:46359)
0924 19:13:48.682710 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:48.682751 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:48.684368 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:48.685122 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:48.685137 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:48.685267 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:48.686304 (+ 1037us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:13:48.686307 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:48.686347 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:13:48.686439 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:48.686996 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:48.687004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:48.687010 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:13:48.687225 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 19:13:48.687471 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
I20260924 19:13:48.689132 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40233
--builtin_ntp_servers=127.11.18.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 19:13:48.690032 21511 heartbeater.cc:347] Connected to a master server at 127.11.18.126:40233
I20260924 19:13:48.690338 21511 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:48.690932 21511 heartbeater.cc:511] Master 127.11.18.126:40233 requested a full tablet report, sending...
I20260924 19:13:48.692566 21194 ts_manager.cc:194] Registered new tserver with Master: f1f4f114ff3d41759a532e40ee03b6be (127.11.18.66:34217)
W20260924 19:13:49.040256 21514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:49.040608 21514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:49.040781 21514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:49.048216 21514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:49.048375 21514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:49.048568 21514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:49.058022 21514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:49.059367 21514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:49.061162 21514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:49.075660 21520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:49.075690 21522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:49.076447 21519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:49.076889 21514 server_base.cc:1034] running on GCE node
I20260924 19:13:49.077590 21514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:49.078547 21514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:49.079823 21514 hybrid_clock.cc:648] HybridClock initialized: now 1790277229079744 us; error 69 us; skew 500 ppm
I20260924 19:13:49.080204 21514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:13:49.083016 21514 webserver.cc:460] Webserver started at http://127.11.18.67:39593/ using document root <none> and password file <none>
I20260924 19:13:49.083791 21514 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:49.083922 21514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:49.084249 21514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:49.087217 21514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "196a12fa1b064a7e9e4d5e54142f73d7"
format_stamp: "Formatted at 2026-09-24 19:13:49 on dist-test-slave-0lgr"
server_key: "ecb31ba4fc8ceef44eb407200bfaf885"
server_key_iv: "9e5aaf202ca3297a22ffcb3ea6daa732"
server_key_version: "encryptionkey@0"
I20260924 19:13:49.088045 21514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "196a12fa1b064a7e9e4d5e54142f73d7"
format_stamp: "Formatted at 2026-09-24 19:13:49 on dist-test-slave-0lgr"
server_key: "ecb31ba4fc8ceef44eb407200bfaf885"
server_key_iv: "9e5aaf202ca3297a22ffcb3ea6daa732"
server_key_version: "encryptionkey@0"
I20260924 19:13:49.094699 21514 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:13:49.098923 21528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:49.100811 21514 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:13:49.101014 21514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "196a12fa1b064a7e9e4d5e54142f73d7"
format_stamp: "Formatted at 2026-09-24 19:13:49 on dist-test-slave-0lgr"
server_key: "ecb31ba4fc8ceef44eb407200bfaf885"
server_key_iv: "9e5aaf202ca3297a22ffcb3ea6daa732"
server_key_version: "encryptionkey@0"
I20260924 19:13:49.101261 21514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:49.107646 21514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:49.108518 21514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:49.108814 21514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:49.109850 21514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:49.111466 21514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:49.111588 21514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:49.111725 21514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:49.111814 21514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:49.149080 21514 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:42875
I20260924 19:13:49.149139 21640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:42875 every 8 connection(s)
I20260924 19:13:49.150694 21514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:49.155426 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 21514
I20260924 19:13:49.155646 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:13:49.156277 11337 external_mini_cluster.cc:1428] Setting key c699318ed6a6c4de649e2d0a21d0d2af
I20260924 19:13:49.160804 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:49.153622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:33583 (local address 127.11.18.126:40233)
0924 19:13:49.153819 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:49.153824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:49.155880 (+ 2056us) server_negotiation.cc:407] Connection header received
0924 19:13:49.158397 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:49.158406 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:49.158478 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:49.158570 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:49.160161 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:49.160174 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:49.160177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:49.160193 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:49.160223 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:49.160234 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:49.160334 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:49.160522 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 19:13:49.160570 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:13:49.162034 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:49.154146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:40233 (local address 127.11.18.67:33583)
0924 19:13:49.155558 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:49.155625 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:49.158047 (+ 2422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:49.158883 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:49.158893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:49.159000 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:49.159819 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:13:49.159823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:49.159866 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:13:49.159960 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:49.160326 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:49.160334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:49.160340 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:13:49.160576 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 19:13:49.160905 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":384,"threads_started":1}
I20260924 19:13:49.163909 21641 heartbeater.cc:347] Connected to a master server at 127.11.18.126:40233
I20260924 19:13:49.164237 21641 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:49.164942 21641 heartbeater.cc:511] Master 127.11.18.126:40233 requested a full tablet report, sending...
I20260924 19:13:49.166569 21194 ts_manager.cc:194] Registered new tserver with Master: 196a12fa1b064a7e9e4d5e54142f73d7 (127.11.18.67:42875)
I20260924 19:13:49.172586 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:13:49.180078 21382 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:13:49.178122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.11.18.126:40233)
0924 19:13:49.178384 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:49.178395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:49.178915 (+ 520us) server_negotiation.cc:407] Connection header received
0924 19:13:49.179063 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:49.179067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:49.179175 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:49.179271 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:49.179793 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44412: BlockingRecv error: recv got EOF from 127.0.0.1:44412 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:13:49.181262 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 21253
I20260924 19:13:49.185698 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:49.182526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.11.18.65:37297)
0924 19:13:49.183090 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:49.183097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:49.183126 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:13:49.183192 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:49.183198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:49.183426 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:49.183562 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:49.184701 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:49.184708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:49.184721 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:49.184740 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:49.184786 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:49.184801 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:49.184917 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:49.185200 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:13:49.185310 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":282,"threads_started":1}
I20260924 19:13:49.187198 21375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:49.273702 21380 heartbeater.cc:503] Master 127.11.18.126:40233 was elected leader, sending a full tablet report...
I20260924 19:13:49.276034 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 21253
I20260924 19:13:49.292877 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 21384
I20260924 19:13:49.297073 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:49.294045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.11.18.66:34217)
0924 19:13:49.294573 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:49.294580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:49.294609 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:13:49.294679 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:49.294686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:49.294983 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:49.295122 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:49.296153 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:49.296162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:49.296175 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:49.296190 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:49.296242 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:49.296266 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:49.296398 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:49.296571 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 19:13:49.296702 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
I20260924 19:13:49.298729 21506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:49.383208 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 21384
I20260924 19:13:49.401728 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 21514
I20260924 19:13:49.405882 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:49.402850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.11.18.67:42875)
0924 19:13:49.403461 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:49.403468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:49.403498 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:13:49.403560 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:49.403567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:49.403789 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:49.403933 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:49.405013 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:49.405019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:49.405031 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:13:49.405043 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:49.405080 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:13:49.405093 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:49.405223 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:49.405499 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 19:13:49.405591 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":302,"threads_started":1}
I20260924 19:13:49.407217 21636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:49.491011 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 21514
I20260924 19:13:49.507671 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 21163
I20260924 19:13:49.509050 21224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:49.612864 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 21163
2026-09-24T19:13:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2471 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:13:49 dist-test-slave-0lgr krb5kdc[21659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:13:49 dist-test-slave-0lgr krb5kdc[21659](info): set up 2 sockets
Sep 24 19:13:49 dist-test-slave-0lgr krb5kdc[21659](info): commencing operation
krb5kdc: starting...
W20260924 19:13:50.762702 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:13:50 dist-test-slave-0lgr krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:13:51.022353 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:34255
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:34255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:51.360961 21672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:51.361377 21672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:51.361550 21672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:51.368947 21672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:13:51.369107 21672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:51.369225 21672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:51.369354 21672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:13:51.369424 21672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:13:51.379663 21672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:34255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:51.381069 21672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:51.382421 21672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:51.395557 21677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:51.395898 21678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:51.395714 21680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:51.396790 21672 server_base.cc:1034] running on GCE node
I20260924 19:13:51.397758 21672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:51.399299 21672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:51.400596 21672 hybrid_clock.cc:648] HybridClock initialized: now 1790277231400534 us; error 52 us; skew 500 ppm
I20260924 19:13:51.401047 21672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:51 dist-test-slave-0lgr krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277231, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:51.411221 21672 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:13:51.412982 21672 webserver.cc:460] Webserver started at http://127.11.18.126:40147/ using document root <none> and password file <none>
I20260924 19:13:51.413825 21672 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:51.413981 21672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:51.414312 21672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:51.417613 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "018b1f6be79c46fe87f713541e1777c7"
format_stamp: "Formatted at 2026-09-24 19:13:51 on dist-test-slave-0lgr"
server_key: "7b265354fddf80739336cc3c13889290"
server_key_iv: "78e70973c982564b8e19183d387f3888"
server_key_version: "encryptionkey@0"
I20260924 19:13:51.418469 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "018b1f6be79c46fe87f713541e1777c7"
format_stamp: "Formatted at 2026-09-24 19:13:51 on dist-test-slave-0lgr"
server_key: "7b265354fddf80739336cc3c13889290"
server_key_iv: "78e70973c982564b8e19183d387f3888"
server_key_version: "encryptionkey@0"
I20260924 19:13:51.424420 21672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 19:13:51.428165 21687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:51.429899 21672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:13:51.430109 21672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "018b1f6be79c46fe87f713541e1777c7"
format_stamp: "Formatted at 2026-09-24 19:13:51 on dist-test-slave-0lgr"
server_key: "7b265354fddf80739336cc3c13889290"
server_key_iv: "78e70973c982564b8e19183d387f3888"
server_key_version: "encryptionkey@0"
I20260924 19:13:51.430342 21672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:51.451970 21672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:51.456838 21672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:51.457162 21672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:51.477782 21672 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:34255
I20260924 19:13:51.477838 21738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:34255 every 8 connection(s)
I20260924 19:13:51.479275 21672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:13:51.483137 21739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:51.485970 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 21672
I20260924 19:13:51.486212 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:13:51.487004 11337 external_mini_cluster.cc:1428] Setting key 510c797ed7f5aa59b91ce61639a2b8ba
I20260924 19:13:51.496398 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7: Bootstrap starting.
I20260924 19:13:51.500013 21739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:51.501070 21739 log.cc:824] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:13:51 dist-test-slave-0lgr krb5kdc[21659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:51.504678 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7: No bootstrap required, opened a new log
I20260924 19:13:51.512730 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:51.489411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.11.18.126:34255)
0924 19:13:51.490252 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:51.490266 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:51.490309 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:13:51.492157 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:51.492184 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:51.492643 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:51.493007 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:51.494953 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:51.496401 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:51.497835 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:51.498583 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:51.498799 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:51.503298 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:51.503312 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:51.503332 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:51.503391 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:51.507565 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:51.508432 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:51.508443 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:51.508454 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:51.508722 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:51.509187 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:51.509192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:51.509196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:51.509992 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:51.510301 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:51.510714 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 19:13:51.511236 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":388,"threads_started":1}
I20260924 19:13:51.515645 21739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018b1f6be79c46fe87f713541e1777c7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34255 } }
I20260924 19:13:51.516073 21739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:51.516189 21739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 018b1f6be79c46fe87f713541e1777c7, State: Initialized, Role: FOLLOWER
I20260924 19:13:51.516776 21739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018b1f6be79c46fe87f713541e1777c7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34255 } }
I20260924 19:13:51.517017 21739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:13:51.517164 21739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:13:51.517367 21739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:51.520139 21739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018b1f6be79c46fe87f713541e1777c7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34255 } }
I20260924 19:13:51.520742 21739 leader_election.cc:302] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 018b1f6be79c46fe87f713541e1777c7; no voters:
I20260924 19:13:51.521528 21739 leader_election.cc:288] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:13:51.521646 21744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:51.522538 21744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [term 1 LEADER]: Becoming Leader. State: Replica: 018b1f6be79c46fe87f713541e1777c7, State: Running, Role: LEADER
I20260924 19:13:51.523187 21744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018b1f6be79c46fe87f713541e1777c7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34255 } }
I20260924 19:13:51.523965 21739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:13:51.525959 21746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 018b1f6be79c46fe87f713541e1777c7. Latest consensus state: current_term: 1 leader_uuid: "018b1f6be79c46fe87f713541e1777c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018b1f6be79c46fe87f713541e1777c7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34255 } } }
I20260924 19:13:51.526331 21746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:51.527451 21751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:13:51.527249 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "018b1f6be79c46fe87f713541e1777c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018b1f6be79c46fe87f713541e1777c7" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34255 } } }
I20260924 19:13:51.528111 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:13:51.534413 21751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:13:51.541216 21761 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:13:51.546357 21751 catalog_manager.cc:1347] Generated new cluster ID: 950f6c26148942dd8958f42c905a155c
I20260924 19:13:51.546492 21751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:13:51.558187 21751 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:13:51.560163 21751 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:13:51.570822 21751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7: Generated new TSK 0
I20260924 19:13:51.571807 21751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:13:51.638574 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34255
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:51.982442 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:51.982949 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:51.983131 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:51.990644 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:51.990828 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:51.991029 21765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:13:52.001426 21765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:52.002926 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:52.004487 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:52.019127 21770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:52.019171 21771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:52.019701 21773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:52.020035 21765 server_base.cc:1034] running on GCE node
I20260924 19:13:52.020737 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:52.021850 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:52.023142 21765 hybrid_clock.cc:648] HybridClock initialized: now 1790277232023100 us; error 36 us; skew 500 ppm
I20260924 19:13:52.023494 21765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:52 dist-test-slave-0lgr krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277232, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:52.026957 21765 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:13:52.028645 21765 webserver.cc:460] Webserver started at http://127.11.18.65:40377/ using document root <none> and password file <none>
I20260924 19:13:52.029462 21765 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:52.029608 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:52.029969 21765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:52.033092 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "cb926211292943f9a29e444869e6fa28"
format_stamp: "Formatted at 2026-09-24 19:13:52 on dist-test-slave-0lgr"
server_key: "c51e6a2ca31df98b008413c5758ce211"
server_key_iv: "dc7a8275ade7b95b8e82063a82966460"
server_key_version: "encryptionkey@0"
I20260924 19:13:52.033988 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "cb926211292943f9a29e444869e6fa28"
format_stamp: "Formatted at 2026-09-24 19:13:52 on dist-test-slave-0lgr"
server_key: "c51e6a2ca31df98b008413c5758ce211"
server_key_iv: "dc7a8275ade7b95b8e82063a82966460"
server_key_version: "encryptionkey@0"
I20260924 19:13:52.039428 21765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:13:52.043232 21780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:52.044775 21765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:13:52.044977 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "cb926211292943f9a29e444869e6fa28"
format_stamp: "Formatted at 2026-09-24 19:13:52 on dist-test-slave-0lgr"
server_key: "c51e6a2ca31df98b008413c5758ce211"
server_key_iv: "dc7a8275ade7b95b8e82063a82966460"
server_key_version: "encryptionkey@0"
I20260924 19:13:52.045202 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:52.064973 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:52.069887 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:52.070215 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:52.071298 21765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:52.072705 21765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:52.072808 21765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:52.072957 21765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:52.073026 21765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:52.109283 21765 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:33845
I20260924 19:13:52.109391 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:33845 every 8 connection(s)
I20260924 19:13:52.110579 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:13:52.115599 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 21765
I20260924 19:13:52.115841 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:13:52.116524 11337 external_mini_cluster.cc:1428] Setting key ef3440068937d3a12aae39ef5fa6c83b
Sep 24 19:13:52 dist-test-slave-0lgr krb5kdc[21659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277232, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:52.132304 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:52.113255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:47373 (local address 127.11.18.126:34255)
0924 19:13:52.113764 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:52.113769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:52.114759 (+ 990us) server_negotiation.cc:407] Connection header received
0924 19:13:52.116733 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:52.116739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:52.116839 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:52.116963 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:52.118574 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:52.119299 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:52.120948 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:52.121412 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:52.121561 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:52.125805 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:52.125811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:52.125814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:52.125850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:52.128558 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:52.129594 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:52.129599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:52.129601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:52.129760 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:52.130294 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:52.130298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:52.130301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:52.130619 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:52.130821 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:52.131854 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0924 19:13:52.132046 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":265,"threads_started":1}
I20260924 19:13:52.133872 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:52.113838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34255 (local address 127.11.18.65:47373)
0924 19:13:52.114535 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:52.114572 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:52.116413 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:52.117226 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:52.117236 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:52.117818 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:52.118392 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:52.118410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:52.119465 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:52.119479 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:52.120786 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:52.120799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:52.121654 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:52.121663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:52.121915 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:52.122683 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:52.122713 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:52.125504 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:52.128730 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:52.128744 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:52.128766 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:52.129453 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:52.129862 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:52.129870 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:52.129876 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:52.130145 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:52.130806 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:52.130818 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:52.131343 (+ 525us) client_negotiation.cc:769] Sending connection context
0924 19:13:52.131674 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 19:13:52.132314 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":277,"threads_started":1}
I20260924 19:13:52.135782 21893 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34255
I20260924 19:13:52.136166 21893 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:52.137214 21893 heartbeater.cc:511] Master 127.11.18.126:34255 requested a full tablet report, sending...
I20260924 19:13:52.139808 21704 ts_manager.cc:194] Registered new tserver with Master: cb926211292943f9a29e444869e6fa28 (127.11.18.65:33845)
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:13:52.141531 21704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:47373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:13:52.173139 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34255
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:52.511994 21899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:52.512286 21899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:52.512382 21899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:52.519793 21899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:52.519922 21899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:52.520040 21899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:13:52.529956 21899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:52.531235 21899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:52.532513 21899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:52.547839 21905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:52.547931 21904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:52.548448 21907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:52.551411 21899 server_base.cc:1034] running on GCE node
I20260924 19:13:52.552148 21899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:52.553267 21899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:52.554557 21899 hybrid_clock.cc:648] HybridClock initialized: now 1790277232554425 us; error 127 us; skew 500 ppm
I20260924 19:13:52.554915 21899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:52 dist-test-slave-0lgr krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277232, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:52.558372 21899 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:13:52.560153 21899 webserver.cc:460] Webserver started at http://127.11.18.66:37253/ using document root <none> and password file <none>
I20260924 19:13:52.560961 21899 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:52.561115 21899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:52.561507 21899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:52.564571 21899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "1c26ab374264409dba2fafedcb8968a5"
format_stamp: "Formatted at 2026-09-24 19:13:52 on dist-test-slave-0lgr"
server_key: "5d98ae1cf3064d29fef811aca845ddee"
server_key_iv: "0075fddbb6b637d809417c173fbdc507"
server_key_version: "encryptionkey@0"
I20260924 19:13:52.565434 21899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "1c26ab374264409dba2fafedcb8968a5"
format_stamp: "Formatted at 2026-09-24 19:13:52 on dist-test-slave-0lgr"
server_key: "5d98ae1cf3064d29fef811aca845ddee"
server_key_iv: "0075fddbb6b637d809417c173fbdc507"
server_key_version: "encryptionkey@0"
I20260924 19:13:52.570822 21899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:13:52.574649 21914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:52.576149 21899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:52.576332 21899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "1c26ab374264409dba2fafedcb8968a5"
format_stamp: "Formatted at 2026-09-24 19:13:52 on dist-test-slave-0lgr"
server_key: "5d98ae1cf3064d29fef811aca845ddee"
server_key_iv: "0075fddbb6b637d809417c173fbdc507"
server_key_version: "encryptionkey@0"
I20260924 19:13:52.576535 21899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:52.599133 21899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:52.604000 21899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:52.604350 21899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:52.605582 21899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:52.607331 21899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:52.607438 21899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:52.607594 21899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:52.607650 21899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:52.644006 21899 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:37383
I20260924 19:13:52.644073 22026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:37383 every 8 connection(s)
I20260924 19:13:52.645468 21899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:13:52.647354 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 21899
I20260924 19:13:52.647572 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:13:52.648145 11337 external_mini_cluster.cc:1428] Setting key 77b28436d92c6703d4d23b86826ff7c4
Sep 24 19:13:52 dist-test-slave-0lgr krb5kdc[21659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277232, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:52.671500 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:52.648665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:49845 (local address 127.11.18.126:34255)
0924 19:13:52.649118 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:52.649124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:52.650599 (+ 1475us) server_negotiation.cc:407] Connection header received
0924 19:13:52.653563 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:52.653567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:52.653689 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:52.653787 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:52.655951 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:52.657085 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:52.659440 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:52.660145 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:52.660253 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:52.664827 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:52.664833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:52.664836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:52.664871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:52.667552 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:52.668417 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:52.668422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:52.668424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:52.668557 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:52.669238 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:52.669243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:52.669245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:52.669625 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:52.670137 (+ 512us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:52.670894 (+ 757us) server_negotiation.cc:299] Negotiation successful
0924 19:13:52.671125 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":226,"threads_started":1}
I20260924 19:13:52.672536 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:52.649291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34255 (local address 127.11.18.66:49845)
0924 19:13:52.650424 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:52.650474 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:52.653210 (+ 2736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:52.654072 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:52.654084 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:52.654888 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:52.655717 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:52.655750 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:52.657266 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:52.657275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:52.659277 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:52.659290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:52.660399 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:52.660408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:52.660764 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:52.661667 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:52.661703 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:52.664521 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:52.667744 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:52.667755 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:52.667776 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:52.668260 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:52.668673 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:52.668677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:52.668681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:52.669053 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:52.670100 (+ 1047us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:52.670109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:52.670508 (+ 399us) client_negotiation.cc:769] Sending connection context
0924 19:13:52.670788 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 19:13:52.671384 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":299,"threads_started":1}
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:13:52.674309 22027 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34255
I20260924 19:13:52.674685 22027 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:13:52.675670 22027 heartbeater.cc:511] Master 127.11.18.126:34255 requested a full tablet report, sending...
I20260924 19:13:52.677524 21704 ts_manager.cc:194] Registered new tserver with Master: 1c26ab374264409dba2fafedcb8968a5 (127.11.18.66:37383)
I20260924 19:13:52.678807 21704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:49845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:13:52.706669 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34255
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:13:53.047302 22033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:13:53.048072 22033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:13:53.048267 22033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:13:53.056082 22033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:13:53.056274 22033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:13:53.056461 22033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:13:53.067608 22033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:34255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:13:53.069072 22033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:13:53.070525 22033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:13:53.085493 22041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:53.085594 22038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:13:53.085621 22039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:13:53.087046 22033 server_base.cc:1034] running on GCE node
I20260924 19:13:53.087816 22033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:13:53.088872 22033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:13:53.090135 22033 hybrid_clock.cc:648] HybridClock initialized: now 1790277233090089 us; error 67 us; skew 500 ppm
I20260924 19:13:53.090497 22033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:13:53 dist-test-slave-0lgr krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277233, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:13:53.094167 22033 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:13:53.095820 22033 webserver.cc:460] Webserver started at http://127.11.18.67:40749/ using document root <none> and password file <none>
I20260924 19:13:53.096534 22033 fs_manager.cc:359] Metadata directory not provided
I20260924 19:13:53.096684 22033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:13:53.096992 22033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:13:53.100209 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "067aba0dbfe44bcf8b129742b67b675a"
format_stamp: "Formatted at 2026-09-24 19:13:53 on dist-test-slave-0lgr"
server_key: "706127cb501dc30aff6550eb4e705919"
server_key_iv: "8b6c8a9442852bffd2f513cd5d2c2d3f"
server_key_version: "encryptionkey@0"
I20260924 19:13:53.101065 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "067aba0dbfe44bcf8b129742b67b675a"
format_stamp: "Formatted at 2026-09-24 19:13:53 on dist-test-slave-0lgr"
server_key: "706127cb501dc30aff6550eb4e705919"
server_key_iv: "8b6c8a9442852bffd2f513cd5d2c2d3f"
server_key_version: "encryptionkey@0"
I20260924 19:13:53.106834 22033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:13:53.110728 22048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:53.112329 22033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:13:53.112641 22033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "067aba0dbfe44bcf8b129742b67b675a"
format_stamp: "Formatted at 2026-09-24 19:13:53 on dist-test-slave-0lgr"
server_key: "706127cb501dc30aff6550eb4e705919"
server_key_iv: "8b6c8a9442852bffd2f513cd5d2c2d3f"
server_key_version: "encryptionkey@0"
I20260924 19:13:53.112879 22033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:13:53.127342 22033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:13:53.132423 22033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:13:53.132772 22033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:13:53.133924 22033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:13:53.135478 22033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:13:53.135598 22033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:53.135771 22033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:13:53.135849 22033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:13:53.145473 21893 heartbeater.cc:503] Master 127.11.18.126:34255 was elected leader, sending a full tablet report...
I20260924 19:13:53.174326 22033 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:43201
I20260924 19:13:53.174374 22160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:43201 every 8 connection(s)
I20260924 19:13:53.175751 22033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:13:53.181463 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 22033
I20260924 19:13:53.181715 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:13:53.182194 11337 external_mini_cluster.cc:1428] Setting key 5a4b0de17a37e920d54f7ac1645a7333
Sep 24 19:13:53 dist-test-slave-0lgr krb5kdc[21659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277233, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:13:53.196569 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.178523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:41359 (local address 127.11.18.126:34255)
0924 19:13:53.178980 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.178985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.179851 (+ 866us) server_negotiation.cc:407] Connection header received
0924 19:13:53.181921 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.181927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.182020 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.182140 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:53.183799 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.184618 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.186260 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.186640 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.186784 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.190367 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:53.190372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:53.190375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:53.190408 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:53.193182 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:53.194050 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:53.194054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:53.194057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:53.194168 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:53.194605 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:53.194609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:53.194611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:53.194941 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:53.195055 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.195792 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.195997 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260924 19:13:53.197788 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.178964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:34255 (local address 127.11.18.67:41359)
0924 19:13:53.179695 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:53.179732 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:53.181625 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:53.182402 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:53.182411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:53.182931 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:13:53.183636 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.183655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.184781 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.184787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.186102 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.186113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.186875 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.186884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.187118 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.187810 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:13:53.187834 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:13:53.190150 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:13:53.193411 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:53.193419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:53.193434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:53.193932 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:53.194275 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:13:53.194279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:13:53.194281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:13:53.194467 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:13:53.195053 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:13:53.195059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:13:53.195392 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 19:13:53.195687 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 19:13:53.196235 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":264,"threads_started":1}
I20260924 19:13:53.199304 22161 heartbeater.cc:347] Connected to a master server at 127.11.18.126:34255
I20260924 19:13:53.199688 22161 heartbeater.cc:464] Registering TS with master...
I20260924 19:13:53.200500 22161 heartbeater.cc:511] Master 127.11.18.126:34255 requested a full tablet report, sending...
I20260924 19:13:53.202083 21704 ts_manager.cc:194] Registered new tserver with Master: 067aba0dbfe44bcf8b129742b67b675a (127.11.18.67:43201)
I20260924 19:13:53.202965 21704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:41359
I20260924 19:13:53.211401 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:13:53.230448 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.216352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.11.18.126:34255)
0924 19:13:53.216549 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.216553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.217040 (+ 487us) server_negotiation.cc:407] Connection header received
0924 19:13:53.217227 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.217231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.217297 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.217441 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:53.219068 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.219881 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.221150 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.221726 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.221841 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.224437 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:53.224442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:53.224445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:53.224479 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:53.227550 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:53.228303 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:53.228308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:53.228310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:53.228434 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:53.228941 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:53.228945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:53.228947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:53.229308 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:53.229638 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.229939 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.230147 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 19:13:53.235548 21704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:13:53.243291 21704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:13:53.288369 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.258806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.11.18.65:33845)
0924 19:13:53.278684 (+ 19878us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.278693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.278727 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:13:53.278837 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.278851 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.279223 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.279531 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:53.280653 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.282198 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.285870 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.286927 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.287111 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.287400 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.287856 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.288049 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19694,"thread_start_us":19523,"threads_started":1}
I20260924 19:13:53.288597 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.258071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36090)
0924 19:13:53.275781 (+ 17710us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:53.275847 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:53.276140 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:53.279598 (+ 3458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:53.279611 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:53.279746 (+ 135us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:53.280272 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.280292 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.282370 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.282380 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.285682 (+ 3302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.285698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.287190 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.287202 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.287472 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.287626 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 19:13:53.287784 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 19:13:53.288020 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17298,"mutex_wait_us":57,"thread_start_us":17065,"threads_started":1}
I20260924 19:13:53.291275 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.259209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:37383 (local address 127.0.0.1:52126)
0924 19:13:53.281822 (+ 22613us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:53.281842 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:53.281969 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:53.282679 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:53.282683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:53.282715 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:53.283310 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.283326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.285105 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.285110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.287360 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.287378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.289921 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.289929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.290136 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.290187 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 19:13:53.290281 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 19:13:53.290427 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22473,"thread_start_us":22311,"threads_started":1}
I20260924 19:13:53.291631 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.259360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.11.18.66:37383)
0924 19:13:53.280226 (+ 20866us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.280235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.282132 (+ 1897us) server_negotiation.cc:407] Connection header received
0924 19:13:53.282238 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.282248 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.282490 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.282688 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:53.283528 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.284956 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.287512 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.289680 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.289853 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.290071 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.290632 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.291282 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20711,"thread_start_us":20587,"threads_started":1}
I20260924 19:13:53.293473 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.259548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:43201 (local address 127.0.0.1:56440)
0924 19:13:53.283431 (+ 23883us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:53.283450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:53.283565 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:53.284161 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:53.284166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:53.284191 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:53.284531 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.284541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.287028 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.287033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.289360 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.289374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.292657 (+ 3283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.292664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.292854 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.292887 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:13:53.292979 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:13:53.293117 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23739,"thread_start_us":23541,"threads_started":1}
I20260924 19:13:53.294250 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.259506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.11.18.67:43201)
0924 19:13:53.278113 (+ 18607us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.278125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.283575 (+ 5450us) server_negotiation.cc:407] Connection header received
0924 19:13:53.283688 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.283695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.283943 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.284160 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:53.284719 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.286860 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.289524 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.292136 (+ 2612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.292302 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.292519 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.293740 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.293924 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18374,"thread_start_us":18175,"threads_started":1}
I20260924 19:13:53.293860 21828 tablet_service.cc:1467] Processing CreateTablet for tablet 0a17aa66037a43bd8e7f932b9e001107 (DEFAULT_TABLE table=test-table [id=7eb63afd807d4d298475ba6870ffd4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:53.295404 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a17aa66037a43bd8e7f932b9e001107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:53.295399 21962 tablet_service.cc:1467] Processing CreateTablet for tablet 0a17aa66037a43bd8e7f932b9e001107 (DEFAULT_TABLE table=test-table [id=7eb63afd807d4d298475ba6870ffd4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:53.296746 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a17aa66037a43bd8e7f932b9e001107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:53.298105 22096 tablet_service.cc:1467] Processing CreateTablet for tablet 0a17aa66037a43bd8e7f932b9e001107 (DEFAULT_TABLE table=test-table [id=7eb63afd807d4d298475ba6870ffd4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:13:53.299492 22096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a17aa66037a43bd8e7f932b9e001107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:13:53.309690 22180 tablet_bootstrap.cc:492] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Bootstrap starting.
I20260924 19:13:53.312325 22181 tablet_bootstrap.cc:492] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: Bootstrap starting.
I20260924 19:13:53.314601 22180 tablet_bootstrap.cc:654] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:53.315924 22181 tablet_bootstrap.cc:654] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:53.315999 22180 log.cc:824] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:53.316257 22182 tablet_bootstrap.cc:492] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Bootstrap starting.
I20260924 19:13:53.317510 22181 log.cc:824] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:53.319859 22182 tablet_bootstrap.cc:654] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Neither blocks nor log segments found. Creating new log.
I20260924 19:13:53.320425 22180 tablet_bootstrap.cc:492] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: No bootstrap required, opened a new log
I20260924 19:13:53.320766 22180 ts_tablet_manager.cc:1397] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260924 19:13:53.320833 22182 log.cc:824] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:13:53.323310 22182 tablet_bootstrap.cc:492] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: No bootstrap required, opened a new log
I20260924 19:13:53.323510 22182 ts_tablet_manager.cc:1397] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 19:13:53.326400 22181 tablet_bootstrap.cc:492] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: No bootstrap required, opened a new log
I20260924 19:13:53.326686 22181 ts_tablet_manager.cc:1397] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20260924 19:13:53.331866 22182 raft_consensus.cc:348] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.332273 22182 raft_consensus.cc:374] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:53.332362 22182 raft_consensus.cc:729] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067aba0dbfe44bcf8b129742b67b675a, State: Initialized, Role: FOLLOWER
I20260924 19:13:53.332075 22180 raft_consensus.cc:348] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.332536 22180 raft_consensus.cc:374] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:53.332639 22180 raft_consensus.cc:729] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c26ab374264409dba2fafedcb8968a5, State: Initialized, Role: FOLLOWER
I20260924 19:13:53.333106 22182 consensus_queue.cc:260] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.333244 22180 consensus_queue.cc:260] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.334362 22161 heartbeater.cc:503] Master 127.11.18.126:34255 was elected leader, sending a full tablet report...
I20260924 19:13:53.335101 22182 ts_tablet_manager.cc:1428] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260924 19:13:53.338105 22027 heartbeater.cc:503] Master 127.11.18.126:34255 was elected leader, sending a full tablet report...
I20260924 19:13:53.338829 22180 ts_tablet_manager.cc:1428] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260924 19:13:53.339725 22181 raft_consensus.cc:348] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.340286 22181 raft_consensus.cc:374] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:13:53.340415 22181 raft_consensus.cc:729] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb926211292943f9a29e444869e6fa28, State: Initialized, Role: FOLLOWER
I20260924 19:13:53.341066 22181 consensus_queue.cc:260] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.342833 22181 ts_tablet_manager.cc:1428] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
W20260924 19:13:53.362980 21894 tablet.cc:2369] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:13:53.397919 22028 tablet.cc:2369] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:13:53.427443 22162 tablet.cc:2369] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:13:53.650373 22187 raft_consensus.cc:482] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:13:53.650689 22187 raft_consensus.cc:504] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.652618 22187 leader_election.cc:288] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb926211292943f9a29e444869e6fa28 (127.11.18.65:33845), 067aba0dbfe44bcf8b129742b67b675a (127.11.18.67:43201)
I20260924 19:13:53.660348 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.653065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:43201 (local address 127.11.18.66:38457)
0924 19:13:53.654244 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:53.654266 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:53.654483 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:53.654828 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:53.654832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:53.654856 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:53.655204 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.655213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.656443 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.656448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.658332 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.658345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.659563 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.659570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.659754 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.659795 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:13:53.659888 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 19:13:53.660022 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":113,"thread_start_us":767,"threads_started":1}
I20260924 19:13:53.660425 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.653422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:38457 (local address 127.11.18.67:43201)
0924 19:13:53.653673 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.653679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.654366 (+ 687us) server_negotiation.cc:407] Connection header received
0924 19:13:53.654633 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.654639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.654709 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.654810 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:53.655346 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.656280 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.658503 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.659355 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.659483 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.659538 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.659959 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.660116 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 19:13:53.660858 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.653136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:42413 (local address 127.11.18.65:33845)
0924 19:13:53.653371 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.653379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.654619 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 19:13:53.654850 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.654856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.654926 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.654982 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:13:53.655826 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.656806 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.658737 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.659676 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.659831 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.659915 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.660462 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.660630 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:13:53.660988 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.653102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.11.18.66:42413)
0924 19:13:53.654511 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:53.654527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:13:53.654714 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:13:53.655187 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:13:53.655192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:13:53.655214 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:13:53.655642 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.655654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.656942 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.656946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.658486 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:13:53.658499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.660034 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.660042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:13:53.660251 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.660290 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:13:53.660413 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 19:13:53.660560 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"mutex_wait_us":2,"thread_start_us":1090,"threads_started":1}
I20260924 19:13:53.661813 22116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a17aa66037a43bd8e7f932b9e001107" candidate_uuid: "1c26ab374264409dba2fafedcb8968a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067aba0dbfe44bcf8b129742b67b675a" is_pre_election: true
I20260924 19:13:53.661933 21848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a17aa66037a43bd8e7f932b9e001107" candidate_uuid: "1c26ab374264409dba2fafedcb8968a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb926211292943f9a29e444869e6fa28" is_pre_election: true
I20260924 19:13:53.662384 22116 raft_consensus.cc:2454] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c26ab374264409dba2fafedcb8968a5 in term 0.
I20260924 19:13:53.662441 21848 raft_consensus.cc:2454] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c26ab374264409dba2fafedcb8968a5 in term 0.
I20260924 19:13:53.663095 21917 leader_election.cc:302] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067aba0dbfe44bcf8b129742b67b675a, 1c26ab374264409dba2fafedcb8968a5; no voters:
I20260924 19:13:53.663550 22187 raft_consensus.cc:2781] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:13:53.663719 22187 raft_consensus.cc:482] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:13:53.663823 22187 raft_consensus.cc:3037] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:53.666306 22187 raft_consensus.cc:504] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.667026 22187 leader_election.cc:288] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [CANDIDATE]: Term 1 election: Requested vote from peers cb926211292943f9a29e444869e6fa28 (127.11.18.65:33845), 067aba0dbfe44bcf8b129742b67b675a (127.11.18.67:43201)
I20260924 19:13:53.667598 22116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a17aa66037a43bd8e7f932b9e001107" candidate_uuid: "1c26ab374264409dba2fafedcb8968a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067aba0dbfe44bcf8b129742b67b675a"
I20260924 19:13:53.667737 21848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a17aa66037a43bd8e7f932b9e001107" candidate_uuid: "1c26ab374264409dba2fafedcb8968a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb926211292943f9a29e444869e6fa28"
I20260924 19:13:53.667840 22116 raft_consensus.cc:3037] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:53.667939 21848 raft_consensus.cc:3037] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:13:53.671317 21848 raft_consensus.cc:2454] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c26ab374264409dba2fafedcb8968a5 in term 1.
I20260924 19:13:53.671317 22116 raft_consensus.cc:2454] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c26ab374264409dba2fafedcb8968a5 in term 1.
I20260924 19:13:53.671895 21918 leader_election.cc:302] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c26ab374264409dba2fafedcb8968a5, cb926211292943f9a29e444869e6fa28; no voters:
I20260924 19:13:53.672377 22187 raft_consensus.cc:2781] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:13:53.672930 22187 raft_consensus.cc:686] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c26ab374264409dba2fafedcb8968a5, State: Running, Role: LEADER
I20260924 19:13:53.673503 22187 consensus_queue.cc:237] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } }
I20260924 19:13:53.678318 21703 catalog_manager.cc:5495] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c26ab374264409dba2fafedcb8968a5 (127.11.18.66). New cstate: current_term: 1 leader_uuid: "1c26ab374264409dba2fafedcb8968a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c26ab374264409dba2fafedcb8968a5" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 37383 } health_report { overall_health: HEALTHY } } }
I20260924 19:13:53.759498 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.752925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.11.18.66:37383)
0924 19:13:53.753222 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.753230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.753506 (+ 276us) server_negotiation.cc:407] Connection header received
0924 19:13:53.753835 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.753839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.753925 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.754096 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:13:53.754498 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.755239 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.756850 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.757514 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.757693 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.758267 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:13:53.758833 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:13:53.759018 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.759080 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.759172 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 19:13:53.768435 22116 raft_consensus.cc:1261] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 1 FOLLOWER]: Refusing update from remote peer 1c26ab374264409dba2fafedcb8968a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:53.768433 21848 raft_consensus.cc:1261] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 1 FOLLOWER]: Refusing update from remote peer 1c26ab374264409dba2fafedcb8968a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:13:53.769588 22191 consensus_queue.cc:1035] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067aba0dbfe44bcf8b129742b67b675a" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:53.770015 22187 consensus_queue.cc:1035] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb926211292943f9a29e444869e6fa28" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 33845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:13:53.783560 22197 mvcc.cc:205] Tried to move back new op lower bound from 7332975549504368640 to 7332975549130321920. Current Snapshot: MvccSnapshot[applied={T|T < 7332975549504368640}]
I20260924 19:13:53.797969 21703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:53.798310 21703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:13:53.802147 21703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 018b1f6be79c46fe87f713541e1777c7: Sending DeleteTablet for 3 replicas of tablet 0a17aa66037a43bd8e7f932b9e001107
I20260924 19:13:53.803582 22096 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a17aa66037a43bd8e7f932b9e001107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:53 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:56440
I20260924 19:13:53.803560 21828 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a17aa66037a43bd8e7f932b9e001107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:53 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:36090
I20260924 19:13:53.803813 21962 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a17aa66037a43bd8e7f932b9e001107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:13:53 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:52126
I20260924 19:13:53.805656 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 21765
I20260924 19:13:53.814306 22206 tablet_replica.cc:330] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: stopping tablet replica
I20260924 19:13:53.815160 22206 raft_consensus.cc:2229] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:13:53.816195 22206 raft_consensus.cc:2258] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:53.817358 22204 tablet_replica.cc:330] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: stopping tablet replica
I20260924 19:13:53.820248 22206 ts_tablet_manager.cc:1905] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:53.820999 22204 raft_consensus.cc:2229] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:53.821836 22204 raft_consensus.cc:2258] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 19:13:53 dist-test-slave-0lgr krb5kdc[21659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:13:53.830564 22204 ts_tablet_manager.cc:1905] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:53.831312 22205 tablet_replica.cc:330] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: stopping tablet replica
I20260924 19:13:53.831851 22205 raft_consensus.cc:2229] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:13:53.832600 22205 raft_consensus.cc:2258] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:13:53.835613 22206 ts_tablet_manager.cc:1918] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:53.835819 22206 log.cc:1196] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/0a17aa66037a43bd8e7f932b9e001107
I20260924 19:13:53.836454 22206 ts_tablet_manager.cc:1939] T 0a17aa66037a43bd8e7f932b9e001107 P 1c26ab374264409dba2fafedcb8968a5: Deleting consensus metadata
I20260924 19:13:53.836892 22205 ts_tablet_manager.cc:1905] T 0a17aa66037a43bd8e7f932b9e001107 P cb926211292943f9a29e444869e6fa28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:13:53.838420 21688 catalog_manager.cc:4841] TS 1c26ab374264409dba2fafedcb8968a5 (127.11.18.66:37383): tablet 0a17aa66037a43bd8e7f932b9e001107 (table test-table [id=7eb63afd807d4d298475ba6870ffd4e3]) successfully deleted
I20260924 19:13:53.838068 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.807291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.11.18.65:33845)
0924 19:13:53.807483 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.807489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.812197 (+ 4708us) server_negotiation.cc:407] Connection header received
0924 19:13:53.812442 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.812449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.812567 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.812685 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:53.818969 (+ 6284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.820198 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.822471 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.823161 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.823266 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.829102 (+ 5836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:53.829124 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:53.829175 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:53.829236 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:53.833673 (+ 4437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:53.834659 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:53.834667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:53.834670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:53.834854 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:53.835655 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:53.835661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:53.835666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:53.836194 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:53.836544 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.837269 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.837623 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:13:53.839643 21888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:53.844447 22204 ts_tablet_manager.cc:1918] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:13:53.844643 22204 log.cc:1196] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/0a17aa66037a43bd8e7f932b9e001107
I20260924 19:13:53.845124 22204 ts_tablet_manager.cc:1939] T 0a17aa66037a43bd8e7f932b9e001107 P 067aba0dbfe44bcf8b129742b67b675a: Deleting consensus metadata
I20260924 19:13:53.847308 21690 catalog_manager.cc:4841] TS 067aba0dbfe44bcf8b129742b67b675a (127.11.18.67:43201): tablet 0a17aa66037a43bd8e7f932b9e001107 (table test-table [id=7eb63afd807d4d298475ba6870ffd4e3]) successfully deleted
I20260924 19:13:53.931538 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 21765
I20260924 19:13:53.949424 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 21899
W20260924 19:13:53.949432 21691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.18.65:33845 (error 108)
I20260924 19:13:53.951802 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.950976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36104)
0924 19:13:53.951434 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:53.951566 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
W20260924 19:13:53.952169 21691 catalog_manager.cc:4584] TS cb926211292943f9a29e444869e6fa28 (127.11.18.65:33845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a17aa66037a43bd8e7f932b9e001107: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Sep 24 19:13:53 dist-test-slave-0lgr krb5kdc[21659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:13:53.967480 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:53.950548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.11.18.66:37383)
0924 19:13:53.950734 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:53.950738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:53.950755 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:13:53.950810 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:53.950813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:53.950871 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:53.950988 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:53.952629 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.953594 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.954856 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:53.955304 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:53.955473 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:53.960492 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:53.960518 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:53.960536 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:53.960580 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:53.964233 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:53.965085 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:53.965094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:53.965097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:53.965255 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:53.965727 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:53.965733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:53.965737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:53.966149 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:53.966495 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:53.966835 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 19:13:53.967074 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:13:53.968327 22022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:54.005846 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:54.005382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36114)
0924 19:13:54.005588 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:54.005697 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 19:13:54.006978 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:54.006554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36122)
0924 19:13:54.006783 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:54.006851 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 19:13:54.067641 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:54.067186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36134)
0924 19:13:54.067401 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:54.067485 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:13:54.068703 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:54.068276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36140)
0924 19:13:54.068419 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:54.068494 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 19:13:54.085530 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 21899
I20260924 19:13:54.104255 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 22033
Sep 24 19:13:54 dist-test-slave-0lgr krb5kdc[21659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:13:54.121877 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:54.105199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.11.18.67:43201)
0924 19:13:54.105406 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:13:54.105412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:13:54.105432 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:13:54.105599 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:13:54.105603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:13:54.105667 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:13:54.105774 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:13:54.107162 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:54.107877 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:54.109236 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:13:54.109750 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:13:54.109894 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:13:54.114323 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:13:54.114338 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:13:54.114355 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:13:54.114423 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:13:54.117865 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:54.119117 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:54.119126 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:54.119128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:54.119316 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:13:54.119921 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:13:54.119930 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:13:54.119934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:13:54.120378 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:13:54.120695 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 19:13:54.121109 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 19:13:54.121453 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:13:54.122653 22156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:54.149444 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:54.148904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36142)
0924 19:13:54.149146 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:54.149239 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 19:13:54.150884 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:54.150349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36150)
0924 19:13:54.150562 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:54.150726 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 19:13:54.217130 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 22033
I20260924 19:13:54.234387 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 21672
I20260924 19:13:54.235757 21734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:13:54.351073 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:13:54.350233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:33845 (local address 127.0.0.1:36162)
0924 19:13:54.350432 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:13:54.350563 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 19:13:54.351634 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 21672
2026-09-24T19:13:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4739 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:13:54 dist-test-slave-0lgr krb5kdc[22216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:13:54 dist-test-slave-0lgr krb5kdc[22216](info): set up 2 sockets
Sep 24 19:13:54 dist-test-slave-0lgr krb5kdc[22216](info): commencing operation
krb5kdc: starting...
W20260924 19:13:55.510859 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:13:55 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:13:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/rangeradmin_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/rangeradmin_127.11.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/rangerlookup_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/rangerlookup_127.11.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/HTTP_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/HTTP_127.11.18.84@KRBTEST.COM.keytab.
I20260924 19:13:55.874025 11337 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJqTBM_/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/postgres -l logfile start
2026-09-24 19:13:58.959 UTC [22246] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:13:58.959 UTC [22246] LOG: listening on IPv4 address "127.11.18.84", port 41591
2026-09-24 19:13:58.963 UTC [22246] LOG: listening on Unix socket "/tmp/.s.PGSQL.41591"
2026-09-24 19:13:58.991 UTC [22249] LOG: database system was shut down at 2026-09-24 19:13:57 UTC
2026-09-24 19:13:58.995 UTC [22246] LOG: database system is ready to accept connections
I20260924 19:14:00.038543 11337 mini_postgres.cc:96] Postgres bound to 41591
2026-09-24 19:14:00.056 UTC [22257] FATAL: database "slave" does not exist
127.11.18.84:41591 - accepting connections
I20260924 19:14:00.057790 11337 mini_ranger.cc:161] Starting Ranger...
I20260924 19:14:00.086875 11337 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:14:00.755304 11337 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:14:00.755491 11337 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-admin
2026-09-24 19:14:01,064 [I] DB FLAVOR :POSTGRES
2026-09-24 19:14:01,065 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:14:01,065 [I] Checking connection..
2026-09-24 19:14:01,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:14:01,337 [I] Checking connection passed.
2026-09-24 19:14:01,338 [I] --------- Verifying version history table ---------
2026-09-24 19:14:01,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:14:01,589 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:14:01,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:14:01,868 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:14:01,868 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:14:01,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:14:02.154 UTC [22321] WARNING: there is no transaction in progress
2026-09-24 19:14:02,162 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:14:02,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:14:02,414 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:14:02,414 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:14:02,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:14:02,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:14:02,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0lgr.c.gcp-upstream.internal','N') ;"
2026-09-24 19:14:03,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:14:03,498 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:14:03,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:14:03,768 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:14:03,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:14:04,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:14:04,308 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:14:04,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:14:05.514 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.533 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.554 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.574 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.594 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.616 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.685 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.695 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.714 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.727 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.740 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.756 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.769 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.778 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.788 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:05.797 UTC [22429] WARNING: there is no transaction in progress
2026-09-24 19:14:06,684 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:14:06,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:14:06,946 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:14:06,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:14:07,204 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:14:07,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:14:07,476 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:14:07,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:14:07,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:14:07,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0lgr.c.gcp-upstream.internal';"
2026-09-24 19:14:08,237 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:14:08,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:14:08,494 [I] DB_PATCHES have already been applied
I20260924 19:14:08.500864 11337 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJqTBM_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:14:08.501000 11337 mini_ranger.cc:193] Using host: 127.11.18.84
I20260924 19:14:08.516693 11337 mini_ranger.cc:227] Ranger admin URL: http://127.11.18.84:41815
Sep 24, 2026 7:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:14:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:14:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:14:09.742017 11337 mini_ranger.cc:160] Time spent starting Ranger: real 9.684s user 0.003s sys 0.041s
Sep 24, 2026 7:14:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:14:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 7:14:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:14:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:14:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:14:33.266557 11337 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/rangerkms_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/rangerkms_127.11.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.18.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.18.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.18.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/HTTP_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.18.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/HTTP_127.11.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:14:33 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277273, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:14:33.441524 11337 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:14:33.463176 11337 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:14:34.679639 11337 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:14:34.679817 11337 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms
I20260924 19:14:34.679952 11337 mini_ranger_kms.cc:227] Using postgres at 127.11.18.84:41591
2026-09-24 19:14:35,113 [I] DB FLAVOR :POSTGRES
2026-09-24 19:14:35,113 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:14:35,113 [I] Checking connection
2026-09-24 19:14:35,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:14:35,389 [I] connection success
2026-09-24 19:14:35,389 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:14:35,389 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:14:35,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:14:35,657 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:14:35,658 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:14:35,658 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:14:35,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:41591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:14:35,935 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:14:36.458879 11337 mini_ranger_kms.cc:310] Created kms service
I20260924 19:14:36.636026 11337 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:14:36.684835 11337 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:14:36 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277273, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:14:37.157822 11337 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:14:37.158135 11337 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:14:37.158205 11337 mini_ranger_kms.cc:240] Using host: 127.11.18.84
I20260924 19:14:37.180194 11337 mini_ranger_kms.cc:276] Ranger KMS PID: 22610
I20260924 19:14:37.180438 11337 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.18.84:56019
Sep 24, 2026 7:14:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:14:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:14:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:14:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:14:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:14:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 19:14:39 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790277279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.18.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:14:39.587498 11337 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.146s user 0.130s sys 0.316s
I20260924 19:14:39.587778 11337 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:14:39.588024 11337 mini_ranger_kms.cc:401] 127.11.18.84:56019/kms/v1/keys
Sep 24 19:14:39 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.18.84@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
Sep 24 19:14:40 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.18.84@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
Sep 24 19:14:42 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:14:42.807433 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41675
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:41675
--ranger_config_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:56019/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:14:43.170743 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:14:43.171044 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:14:43.171217 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:14:43.180429 22660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:14:43.180589 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:14:43.180711 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:14:43.180819 22660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:14:43.180908 22660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:14:43.192853 22660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.18.84:56019/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:41675
--ranger_config_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:14:43.194412 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:14:43.196066 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:14:43.211297 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:14:43.211536 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:14:43.211817 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:14:43.211892 22660 server_base.cc:1034] running on GCE node
I20260924 19:14:43.212824 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:14:43.214564 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:14:43.215785 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790277283215735 us; error 46 us; skew 500 ppm
I20260924 19:14:43.216217 22660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:14:43 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:14:43.224855 22660 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:14:43.227003 22660 webserver.cc:460] Webserver started at http://127.11.18.126:37635/ using document root <none> and password file <none>
I20260924 19:14:43.227799 22660 fs_manager.cc:359] Metadata directory not provided
I20260924 19:14:43.227957 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:14:43.228336 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:14:43 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:14:43.283595 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "c8369aa8461348bface45e48f2e11206"
format_stamp: "Formatted at 2026-09-24 19:14:43 on dist-test-slave-0lgr"
server_key: "0fe73b7e69560a8239ded46ff0a5a0be"
server_key_iv: "b6d6d1712e828822158dcb775bcb6a24"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:43.284623 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "c8369aa8461348bface45e48f2e11206"
format_stamp: "Formatted at 2026-09-24 19:14:43 on dist-test-slave-0lgr"
server_key: "0fe73b7e69560a8239ded46ff0a5a0be"
server_key_iv: "b6d6d1712e828822158dcb775bcb6a24"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:43.291358 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 19:14:43.308629 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:43.310290 22660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:14:43.310497 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "c8369aa8461348bface45e48f2e11206"
format_stamp: "Formatted at 2026-09-24 19:14:43 on dist-test-slave-0lgr"
server_key: "0fe73b7e69560a8239ded46ff0a5a0be"
server_key_iv: "b6d6d1712e828822158dcb775bcb6a24"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:43.310747 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:14:43.351711 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:14:43.356170 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:14:43.356518 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:14:43.377079 22660 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:41675
I20260924 19:14:43.377097 22726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:41675 every 8 connection(s)
I20260924 19:14:43.378509 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:14:43.379765 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 22660
I20260924 19:14:43.379988 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:14:43.381006 11337 external_mini_cluster.cc:1428] Setting key 25cd1154437c20a813f4fe45da8f8a94
I20260924 19:14:43.384058 22727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:14:43.400658 22727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206: Bootstrap starting.
Sep 24 19:14:43 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:14:43.405673 22727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206: Neither blocks nor log segments found. Creating new log.
I20260924 19:14:43.407157 22727 log.cc:824] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206: Log is configured to *not* fsync() on all Append() calls
I20260924 19:14:43.411427 22727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206: No bootstrap required, opened a new log
I20260924 19:14:43.414004 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:43.383775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.11.18.126:41675)
0924 19:14:43.393962 (+ 10187us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:43.393975 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:43.394050 (+ 75us) server_negotiation.cc:407] Connection header received
0924 19:14:43.395829 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:43.395862 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:43.396288 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:43.396723 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:14:43.398138 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:43.399431 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:43.400687 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:43.401618 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:43.401807 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:43.406107 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:14:43.406115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:14:43.406132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:14:43.406182 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:14:43.409691 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:43.410452 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:43.410458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:43.410465 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:43.410607 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:43.411024 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:43.411028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:43.411030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:43.411585 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:14:43.411883 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:43.412371 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 19:14:43.412806 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9922,"thread_start_us":9684,"threads_started":1}
I20260924 19:14:43.422837 22727 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8369aa8461348bface45e48f2e11206" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41675 } }
I20260924 19:14:43.423355 22727 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:14:43.423501 22727 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8369aa8461348bface45e48f2e11206, State: Initialized, Role: FOLLOWER
I20260924 19:14:43.424213 22727 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8369aa8461348bface45e48f2e11206" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41675 } }
I20260924 19:14:43.424487 22727 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:14:43.424626 22727 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:14:43.424830 22727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:14:43.428234 22727 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8369aa8461348bface45e48f2e11206" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41675 } }
I20260924 19:14:43.428915 22727 leader_election.cc:302] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8369aa8461348bface45e48f2e11206; no voters:
I20260924 19:14:43.429687 22727 leader_election.cc:288] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:14:43.429844 22732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:14:43.431586 22732 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [term 1 LEADER]: Becoming Leader. State: Replica: c8369aa8461348bface45e48f2e11206, State: Running, Role: LEADER
I20260924 19:14:43.432408 22727 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:14:43.432314 22732 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8369aa8461348bface45e48f2e11206" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41675 } }
I20260924 19:14:43.436309 22734 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8369aa8461348bface45e48f2e11206. Latest consensus state: current_term: 1 leader_uuid: "c8369aa8461348bface45e48f2e11206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8369aa8461348bface45e48f2e11206" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41675 } } }
I20260924 19:14:43.436221 22733 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8369aa8461348bface45e48f2e11206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8369aa8461348bface45e48f2e11206" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41675 } } }
I20260924 19:14:43.436717 22733 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [sys.catalog]: This master's current role is: LEADER
I20260924 19:14:43.436717 22734 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206 [sys.catalog]: This master's current role is: LEADER
I20260924 19:14:43.446988 22739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:14:43.457229 22739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:14:43.469717 22739 catalog_manager.cc:1347] Generated new cluster ID: 331f4b2911574c94a2c0c0e0434b5755
I20260924 19:14:43.469875 22739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:14:43.492307 22739 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:14:43.495409 22739 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:14:43.515444 22727 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:14:43.518335 22739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206: Generated new TSK 0
I20260924 19:14:43.519775 22739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:14:44.421809 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:56019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41675
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:14:44 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790277284, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:14:44.889423 22772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:14:44.889873 22772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:14:44.890051 22772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:14:44.903879 22772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:14:44.904052 22772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:14:44.904317 22772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:14:44.922613 22772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.18.84:56019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:14:44.924469 22772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:14:44.926477 22772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:14:44.947626 22782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:14:44.947584 22783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:14:44.954722 22785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:14:44.955484 22772 server_base.cc:1034] running on GCE node
I20260924 19:14:44.956437 22772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:14:44.960520 22772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:14:44.961907 22772 hybrid_clock.cc:648] HybridClock initialized: now 1790277284961846 us; error 95 us; skew 500 ppm
I20260924 19:14:44.962415 22772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:14:44 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277284, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:14:44.968108 22772 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:14:44.971132 22772 webserver.cc:460] Webserver started at http://127.11.18.65:38471/ using document root <none> and password file <none>
I20260924 19:14:44.972275 22772 fs_manager.cc:359] Metadata directory not provided
I20260924 19:14:44.972471 22772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:14:44.972931 22772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:14:44 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277284, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:14:44.994580 22772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "9460271a594e47168138112388d1aeaa"
format_stamp: "Formatted at 2026-09-24 19:14:44 on dist-test-slave-0lgr"
server_key: "133f1fd07a8e41779bf07437c16402af"
server_key_iv: "dc593724449d7e184cb8872e5b9cc3a9"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:44.996153 22772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "9460271a594e47168138112388d1aeaa"
format_stamp: "Formatted at 2026-09-24 19:14:44 on dist-test-slave-0lgr"
server_key: "133f1fd07a8e41779bf07437c16402af"
server_key_iv: "dc593724449d7e184cb8872e5b9cc3a9"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:45.007509 22772 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.007s sys 0.003s
I20260924 19:14:45.029703 22792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:45.031481 22772 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260924 19:14:45.031697 22772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "9460271a594e47168138112388d1aeaa"
format_stamp: "Formatted at 2026-09-24 19:14:44 on dist-test-slave-0lgr"
server_key: "133f1fd07a8e41779bf07437c16402af"
server_key_iv: "dc593724449d7e184cb8872e5b9cc3a9"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:45.031953 22772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:14:45.054987 22772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:14:45.061108 22772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:14:45.061592 22772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:14:45.069139 22772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:14:45.078027 22772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:14:45.078248 22772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 19:14:45.078471 22772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:14:45.078611 22772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:45.243305 22904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:44643 every 8 connection(s)
I20260924 19:14:45.243422 22772 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:44643
I20260924 19:14:45.245316 22772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:14:45.252350 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 22772
I20260924 19:14:45.252578 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:14:45.253476 11337 external_mini_cluster.cc:1428] Setting key 391535fa50a46b5db1da5e1deb4e2885
Sep 24 19:14:45 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277284, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:14:45.288861 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:45.250098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:37197 (local address 127.11.18.126:41675)
0924 19:14:45.262897 (+ 12799us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:45.262904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:45.262925 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:14:45.262989 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:45.262995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:45.263092 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:45.263208 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:14:45.265638 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:45.266942 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:45.271359 (+ 4417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:45.272163 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:45.272343 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:45.279763 (+ 7420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:14:45.279772 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:14:45.279778 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:14:45.279835 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:14:45.284103 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:45.285452 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:45.285459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:45.285463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:45.285649 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:45.286328 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:45.286336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:45.286343 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:45.286769 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:14:45.286920 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:45.288115 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0924 19:14:45.288402 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12652,"thread_start_us":12537,"threads_started":1}
I20260924 19:14:45.291415 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:45.249373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41675 (local address 127.11.18.65:37197)
0924 19:14:45.258301 (+ 8928us) negotiation.cc:107] Waiting for socket to connect
0924 19:14:45.258360 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 19:14:45.261975 (+ 3615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:14:45.263724 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:14:45.263739 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:14:45.264443 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:14:45.265386 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:45.265414 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:45.267196 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:45.267202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:45.271118 (+ 3916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:45.271145 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:45.272772 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:45.272783 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:45.273168 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:45.274235 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:14:45.274272 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:14:45.277993 (+ 3721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:14:45.284404 (+ 6411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:14:45.284416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:14:45.284439 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:14:45.285235 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:14:45.285831 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:14:45.285836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:14:45.285840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:14:45.286165 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:14:45.287050 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:14:45.287059 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:14:45.287582 (+ 523us) client_negotiation.cc:769] Sending connection context
0924 19:14:45.289141 (+ 1559us) client_negotiation.cc:240] Negotiation successful
0924 19:14:45.289813 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8571,"thread_start_us":8330,"threads_started":1}
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:14:45.294035 22905 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41675
I20260924 19:14:45.294411 22905 heartbeater.cc:464] Registering TS with master...
I20260924 19:14:45.295395 22905 heartbeater.cc:511] Master 127.11.18.126:41675 requested a full tablet report, sending...
I20260924 19:14:45.298300 22692 ts_manager.cc:194] Registered new tserver with Master: 9460271a594e47168138112388d1aeaa (127.11.18.65:44643)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
I20260924 19:14:45.300734 22692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:37197
Sep 24 19:14:45 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277284, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:14:45.340133 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:56019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41675
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:14:45 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277284, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
W20260924 19:14:45.888654 22911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:14:45.889225 22911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:14:45.889468 22911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:14:45.901163 22911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:14:45.901453 22911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:14:45.901705 22911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:14:45.918423 22911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.18.84:56019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:14:45.920051 22911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:14:45.921752 22911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:14:45.955827 22919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:14:45.959115 22921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:14:45.959304 22918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:14:45.960754 22911 server_base.cc:1034] running on GCE node
I20260924 19:14:45.961602 22911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:14:45.969189 22911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:14:45.970595 22911 hybrid_clock.cc:648] HybridClock initialized: now 1790277285970512 us; error 83 us; skew 500 ppm
I20260924 19:14:45.971050 22911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:14:45 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277285, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:14:45.974848 22911 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:14:45.977872 22911 webserver.cc:460] Webserver started at http://127.11.18.66:35631/ using document root <none> and password file <none>
I20260924 19:14:45.978675 22911 fs_manager.cc:359] Metadata directory not provided
I20260924 19:14:45.978840 22911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:14:45.979243 22911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:14:45 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277285, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:14:46.008562 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "d48f6cf8da014fb7aa70abce3b592bb9"
format_stamp: "Formatted at 2026-09-24 19:14:46 on dist-test-slave-0lgr"
server_key: "aa68ed0b4b86c46dbe3104170b817d2f"
server_key_iv: "7714a772c86f40eb00726cad4f749b77"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:46.009594 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "d48f6cf8da014fb7aa70abce3b592bb9"
format_stamp: "Formatted at 2026-09-24 19:14:46 on dist-test-slave-0lgr"
server_key: "aa68ed0b4b86c46dbe3104170b817d2f"
server_key_iv: "7714a772c86f40eb00726cad4f749b77"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:46.026822 22911 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.006s sys 0.009s
I20260924 19:14:46.046872 22935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:46.048831 22911 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:14:46.049121 22911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "d48f6cf8da014fb7aa70abce3b592bb9"
format_stamp: "Formatted at 2026-09-24 19:14:46 on dist-test-slave-0lgr"
server_key: "aa68ed0b4b86c46dbe3104170b817d2f"
server_key_iv: "7714a772c86f40eb00726cad4f749b77"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:46.049503 22911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:14:46.090545 22911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:14:46.104065 22911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:14:46.104453 22911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:14:46.105710 22911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:14:46.107519 22911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:14:46.107618 22911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:46.107792 22911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:14:46.107853 22911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:46.151713 22911 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:40351
I20260924 19:14:46.151806 23047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:40351 every 8 connection(s)
I20260924 19:14:46.153396 22911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:14:46.163090 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 22911
I20260924 19:14:46.163280 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:14:46.163838 11337 external_mini_cluster.cc:1428] Setting key 8042c72161acee47941b2e3d21ab5705
Sep 24 19:14:46 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277285, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:14:46.182502 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.156704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:57937 (local address 127.11.18.126:41675)
0924 19:14:46.157226 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:46.157232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:46.158469 (+ 1237us) server_negotiation.cc:407] Connection header received
0924 19:14:46.161475 (+ 3006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:46.161479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:46.161570 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:46.161672 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:14:46.163636 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.164686 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.167056 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.167842 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.167998 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.173268 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:14:46.173273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:14:46.173277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:14:46.173315 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:14:46.177915 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:46.179374 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:46.179384 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:46.179390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:46.179564 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:46.180130 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:46.180138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:46.180144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:46.180561 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:14:46.180721 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:46.181710 (+ 989us) server_negotiation.cc:299] Negotiation successful
0924 19:14:46.182003 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":242,"threads_started":1}
I20260924 19:14:46.183605 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.157471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41675 (local address 127.11.18.66:57937)
0924 19:14:46.158269 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0924 19:14:46.158331 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 19:14:46.161118 (+ 2787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:14:46.161960 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:14:46.161973 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:14:46.162593 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:14:46.163418 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.163441 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.164859 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.164865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.166896 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.166911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.168088 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.168095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.168388 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.169215 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:14:46.169245 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:14:46.172992 (+ 3747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:14:46.178159 (+ 5167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:14:46.178173 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:14:46.178199 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:14:46.179139 (+ 940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:14:46.179705 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:14:46.179711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:14:46.179715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:14:46.179984 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:14:46.180731 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:14:46.180740 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:14:46.181200 (+ 460us) client_negotiation.cc:769] Sending connection context
0924 19:14:46.181559 (+ 359us) client_negotiation.cc:240] Negotiation successful
0924 19:14:46.182101 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":303,"threads_started":1}
I20260924 19:14:46.185722 23048 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41675
I20260924 19:14:46.186090 23048 heartbeater.cc:464] Registering TS with master...
I20260924 19:14:46.187004 23048 heartbeater.cc:511] Master 127.11.18.126:41675 requested a full tablet report, sending...
I20260924 19:14:46.188827 22692 ts_manager.cc:194] Registered new tserver with Master: d48f6cf8da014fb7aa70abce3b592bb9 (127.11.18.66:40351)
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:14:46.190402 22692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:57937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:14:46.226325 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:56019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41675
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:14:46.305301 22905 heartbeater.cc:503] Master 127.11.18.126:41675 was elected leader, sending a full tablet report...
W20260924 19:14:46.619501 23054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:14:46.619902 23054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:14:46.620091 23054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:14:46.628093 23054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:14:46.628222 23054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:14:46.628336 23054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:14:46.639052 23054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.18.84:56019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:14:46.640376 23054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:14:46.641754 23054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:14:46.656845 23060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:14:46.656993 23059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:14:46.657027 23062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:14:46.658354 23054 server_base.cc:1034] running on GCE node
I20260924 19:14:46.659046 23054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:14:46.660317 23054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:14:46.661621 23054 hybrid_clock.cc:648] HybridClock initialized: now 1790277286661572 us; error 80 us; skew 500 ppm
I20260924 19:14:46.662123 23054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:14:46 dist-test-slave-0lgr krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277286, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:14:46.665838 23054 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:14:46.667783 23054 webserver.cc:460] Webserver started at http://127.11.18.67:37029/ using document root <none> and password file <none>
I20260924 19:14:46.668568 23054 fs_manager.cc:359] Metadata directory not provided
I20260924 19:14:46.668733 23054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:14:46.669083 23054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:14:46 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277286, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:14:46.686084 23054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "44d06693b42f4c97a03bb8de1a551df7"
format_stamp: "Formatted at 2026-09-24 19:14:46 on dist-test-slave-0lgr"
server_key: "d29a8d95e0db2a055b80e026f439155c"
server_key_iv: "af8539aeab3f6374c1c707136f8bc8ac"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:46.687072 23054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "44d06693b42f4c97a03bb8de1a551df7"
format_stamp: "Formatted at 2026-09-24 19:14:46 on dist-test-slave-0lgr"
server_key: "d29a8d95e0db2a055b80e026f439155c"
server_key_iv: "af8539aeab3f6374c1c707136f8bc8ac"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:46.693485 23054 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:14:46.712987 23069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:46.714833 23054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:14:46.715054 23054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "44d06693b42f4c97a03bb8de1a551df7"
format_stamp: "Formatted at 2026-09-24 19:14:46 on dist-test-slave-0lgr"
server_key: "d29a8d95e0db2a055b80e026f439155c"
server_key_iv: "af8539aeab3f6374c1c707136f8bc8ac"
server_key_version: "kuduclusterkey@0"
I20260924 19:14:46.715355 23054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:14:46.733291 23054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:14:46.737869 23054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:14:46.738178 23054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:14:46.739235 23054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:14:46.740679 23054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:14:46.740752 23054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:46.740885 23054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:14:46.740933 23054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:14:46.782678 23054 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:46837
I20260924 19:14:46.782704 23181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:46837 every 8 connection(s)
I20260924 19:14:46.784044 23054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:14:46.791643 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 23054
I20260924 19:14:46.791826 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:14:46.792341 11337 external_mini_cluster.cc:1428] Setting key f8b0a7bfcaf1002f71aaca0cde133f76
Sep 24 19:14:46 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277286, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:14:46.806919 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.786839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:59353 (local address 127.11.18.126:41675)
0924 19:14:46.787405 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:46.787410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:46.788463 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 19:14:46.790338 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:46.790343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:46.790438 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:46.790561 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:14:46.792094 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.793119 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.795337 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.795930 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.796164 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.799834 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:14:46.799839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:14:46.799842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:14:46.799878 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:14:46.803344 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:46.804226 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:46.804232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:46.804238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:46.804384 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:46.804889 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:46.804896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:46.804905 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:46.805212 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:14:46.805456 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:46.806218 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 19:14:46.806504 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":299,"threads_started":1}
I20260924 19:14:46.808287 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.787531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41675 (local address 127.11.18.67:59353)
0924 19:14:46.788294 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0924 19:14:46.788332 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:14:46.790060 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:14:46.790798 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:14:46.790807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:14:46.791240 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:14:46.791907 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.791941 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.793279 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.793285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.795191 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.795203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.796244 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.796253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.796522 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.797220 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:14:46.797242 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:14:46.799598 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:14:46.803528 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:14:46.803540 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:14:46.803561 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:14:46.804076 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:14:46.804508 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:14:46.804514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:14:46.804519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:14:46.804753 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:14:46.805425 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:14:46.805434 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:14:46.805823 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 19:14:46.806117 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 19:14:46.806689 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":325,"threads_started":1}
I20260924 19:14:46.810077 23182 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41675
I20260924 19:14:46.810438 23182 heartbeater.cc:464] Registering TS with master...
I20260924 19:14:46.811158 23182 heartbeater.cc:511] Master 127.11.18.126:41675 requested a full tablet report, sending...
I20260924 19:14:46.812744 22692 ts_manager.cc:194] Registered new tserver with Master: 44d06693b42f4c97a03bb8de1a551df7 (127.11.18.67:46837)
I20260924 19:14:46.813742 22692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:59353
I20260924 19:14:46.820847 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:14:46.841629 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.825953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.11.18.126:41675)
0924 19:14:46.826150 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:46.826155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:46.826695 (+ 540us) server_negotiation.cc:407] Connection header received
0924 19:14:46.826841 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:46.826845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:46.826940 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:46.827042 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:14:46.828722 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.829643 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.831185 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.831689 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.831793 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.835495 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:14:46.835500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:14:46.835504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:14:46.835543 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:14:46.838701 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:46.839586 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:46.839590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:46.839592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:46.839723 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:46.840263 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:46.840267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:46.840270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:46.840574 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:14:46.840761 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:46.841130 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 19:14:46.841301 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:14:46.846402 22692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:14:46.853744 22692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:14:46.881139 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.868214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46837 (local address 127.0.0.1:51838)
0924 19:14:46.870636 (+ 2422us) negotiation.cc:107] Waiting for socket to connect
0924 19:14:46.870664 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:14:46.870785 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:14:46.871850 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:14:46.871864 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:14:46.871910 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:14:46.874088 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.874110 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.876115 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.876125 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.878510 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.878522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.879980 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.879987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.880264 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.880320 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 19:14:46.880466 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 19:14:46.880680 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2277,"mutex_wait_us":81,"thread_start_us":2099,"threads_started":1}
I20260924 19:14:46.882194 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.868626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.11.18.67:46837)
0924 19:14:46.870968 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:46.870981 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:46.871028 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:14:46.871156 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:46.871168 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:46.871530 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:46.872572 (+ 1042us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:14:46.874452 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.875953 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.878673 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.879734 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.879914 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.880172 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:46.881606 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0924 19:14:46.881812 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1462,"thread_start_us":1306,"threads_started":1}
I20260924 19:14:46.882819 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.868288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.11.18.65:44643)
0924 19:14:46.871329 (+ 3041us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:46.871340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:46.871374 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:14:46.871509 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:46.871519 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:46.871787 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:46.872579 (+ 792us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:14:46.873571 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.875456 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.878726 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.880301 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.880462 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.880640 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:46.881809 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0924 19:14:46.881998 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2857,"thread_start_us":2717,"threads_started":1}
I20260924 19:14:46.881659 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.868026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:44643 (local address 127.0.0.1:36496)
0924 19:14:46.869019 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0924 19:14:46.869075 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 19:14:46.869292 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:14:46.872044 (+ 2752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:14:46.872056 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:14:46.872093 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:14:46.873206 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.873222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.875635 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.875641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.878513 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.878527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.880539 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.880545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.880712 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.880742 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:14:46.881089 (+ 347us) client_negotiation.cc:240] Negotiation successful
0924 19:14:46.881253 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":32,"thread_start_us":920,"threads_started":1}
I20260924 19:14:46.883473 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.870492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:40351 (local address 127.0.0.1:41438)
0924 19:14:46.874547 (+ 4055us) negotiation.cc:107] Waiting for socket to connect
0924 19:14:46.874568 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:14:46.874697 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:14:46.875451 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:14:46.875457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:14:46.875483 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:14:46.876001 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.876012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.878013 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.878020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.880309 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:46.880323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.882490 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.882496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:46.882663 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.882691 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 19:14:46.882787 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:14:46.883112 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3923,"mutex_wait_us":36,"thread_start_us":3535,"threads_started":1}
I20260924 19:14:46.883683 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:46.870574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.11.18.66:40351)
0924 19:14:46.873430 (+ 2856us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:46.873452 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:46.874927 (+ 1475us) server_negotiation.cc:407] Connection header received
0924 19:14:46.875030 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:46.875036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:46.875240 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:46.875451 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:14:46.876229 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.877806 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.880724 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:46.882207 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:46.882380 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:46.882569 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:46.883135 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 19:14:46.883319 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2630,"thread_start_us":2495,"threads_started":1}
I20260924 19:14:46.887143 22840 tablet_service.cc:1467] Processing CreateTablet for tablet bf128a49298e4b0baa107a6770985c0d (DEFAULT_TABLE table=test-table [id=90d078200024413ea1dc582c7a8a0305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:14:46.887192 22983 tablet_service.cc:1467] Processing CreateTablet for tablet bf128a49298e4b0baa107a6770985c0d (DEFAULT_TABLE table=test-table [id=90d078200024413ea1dc582c7a8a0305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:14:46.887192 23117 tablet_service.cc:1467] Processing CreateTablet for tablet bf128a49298e4b0baa107a6770985c0d (DEFAULT_TABLE table=test-table [id=90d078200024413ea1dc582c7a8a0305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:14:46.888589 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf128a49298e4b0baa107a6770985c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:14:46.888638 22983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf128a49298e4b0baa107a6770985c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:14:46.888641 23117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf128a49298e4b0baa107a6770985c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:14:46.901666 23201 tablet_bootstrap.cc:492] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Bootstrap starting.
I20260924 19:14:46.906468 23203 tablet_bootstrap.cc:492] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Bootstrap starting.
I20260924 19:14:46.906790 23201 tablet_bootstrap.cc:654] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Neither blocks nor log segments found. Creating new log.
I20260924 19:14:46.908311 23201 log.cc:824] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Log is configured to *not* fsync() on all Append() calls
I20260924 19:14:46.908376 23202 tablet_bootstrap.cc:492] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Bootstrap starting.
I20260924 19:14:46.910617 23203 tablet_bootstrap.cc:654] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Neither blocks nor log segments found. Creating new log.
I20260924 19:14:46.911981 23203 log.cc:824] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:14:46.912361 23202 tablet_bootstrap.cc:654] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Neither blocks nor log segments found. Creating new log.
I20260924 19:14:46.912501 23201 tablet_bootstrap.cc:492] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: No bootstrap required, opened a new log
I20260924 19:14:46.912899 23201 ts_tablet_manager.cc:1397] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260924 19:14:46.913949 23202 log.cc:824] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:14:46.915612 23203 tablet_bootstrap.cc:492] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: No bootstrap required, opened a new log
I20260924 19:14:46.915911 23203 ts_tablet_manager.cc:1397] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 19:14:46.917600 23202 tablet_bootstrap.cc:492] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: No bootstrap required, opened a new log
I20260924 19:14:46.917897 23202 ts_tablet_manager.cc:1397] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 19:14:46.921648 23201 raft_consensus.cc:348] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:46.922045 23201 raft_consensus.cc:374] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:14:46.922194 23201 raft_consensus.cc:729] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9460271a594e47168138112388d1aeaa, State: Initialized, Role: FOLLOWER
I20260924 19:14:46.922840 23201 consensus_queue.cc:260] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:46.924788 23201 ts_tablet_manager.cc:1428] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260924 19:14:46.928414 23203 raft_consensus.cc:348] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:46.929383 23203 raft_consensus.cc:374] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:14:46.929533 23203 raft_consensus.cc:729] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d06693b42f4c97a03bb8de1a551df7, State: Initialized, Role: FOLLOWER
I20260924 19:14:46.930284 23203 consensus_queue.cc:260] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:46.930670 23202 raft_consensus.cc:348] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:46.931180 23202 raft_consensus.cc:374] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:14:46.931286 23202 raft_consensus.cc:729] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48f6cf8da014fb7aa70abce3b592bb9, State: Initialized, Role: FOLLOWER
I20260924 19:14:46.931528 23182 heartbeater.cc:503] Master 127.11.18.126:41675 was elected leader, sending a full tablet report...
I20260924 19:14:46.931903 23202 consensus_queue.cc:260] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:46.932322 23203 ts_tablet_manager.cc:1428] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260924 19:14:46.933133 23048 heartbeater.cc:503] Master 127.11.18.126:41675 was elected leader, sending a full tablet report...
I20260924 19:14:46.933756 23202 ts_tablet_manager.cc:1428] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260924 19:14:47.004819 22906 tablet.cc:2369] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:14:47.035903 23183 tablet.cc:2369] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:14:47.082163 23208 raft_consensus.cc:482] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:14:47.082541 23208 raft_consensus.cc:504] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:47.084322 23208 leader_election.cc:288] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9460271a594e47168138112388d1aeaa (127.11.18.65:44643), d48f6cf8da014fb7aa70abce3b592bb9 (127.11.18.66:40351)
I20260924 19:14:47.093748 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:47.084832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:53083 (local address 127.11.18.65:44643)
0924 19:14:47.085069 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:47.085078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:47.085213 (+ 135us) server_negotiation.cc:407] Connection header received
0924 19:14:47.085491 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:47.085498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:47.085589 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:47.085704 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:14:47.086182 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.087521 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.091102 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.092587 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.092792 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:47.092875 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:47.093359 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 19:14:47.093498 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:14:47.093690 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:47.084870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:40885 (local address 127.11.18.66:40351)
0924 19:14:47.085087 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:47.085095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:47.085823 (+ 728us) server_negotiation.cc:407] Connection header received
0924 19:14:47.085979 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:47.085982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:47.086042 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:47.086170 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:14:47.086665 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.087668 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.091095 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.092587 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.092787 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:47.092878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:47.093241 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 19:14:47.093410 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:14:47.093797 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:47.084864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:40351 (local address 127.11.18.67:40885)
0924 19:14:47.085706 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 19:14:47.085722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:14:47.085827 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:14:47.086177 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:14:47.086180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:14:47.086198 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:14:47.086477 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:47.086485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.087838 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.087845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:47.090850 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:47.090870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.092825 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.092829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:47.093007 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:47.093051 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:14:47.093163 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:14:47.093311 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":105,"thread_start_us":517,"threads_started":1}
I20260924 19:14:47.093824 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:47.084731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:44643 (local address 127.11.18.67:53083)
0924 19:14:47.085086 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 19:14:47.085105 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:14:47.085259 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:14:47.085734 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:14:47.085743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:14:47.085783 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:14:47.086019 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:47.086028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.087713 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.087717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:47.090850 (+ 3133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:14:47.090870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.092800 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.092812 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:14:47.093085 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:47.093127 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:14:47.093204 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:14:47.093311 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":102,"spinlock_wait_cycles":23808}
I20260924 19:14:47.094901 23003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf128a49298e4b0baa107a6770985c0d" candidate_uuid: "44d06693b42f4c97a03bb8de1a551df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" is_pre_election: true
I20260924 19:14:47.095294 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf128a49298e4b0baa107a6770985c0d" candidate_uuid: "44d06693b42f4c97a03bb8de1a551df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9460271a594e47168138112388d1aeaa" is_pre_election: true
I20260924 19:14:47.095638 23003 raft_consensus.cc:2454] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d06693b42f4c97a03bb8de1a551df7 in term 0.
I20260924 19:14:47.095813 22860 raft_consensus.cc:2454] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d06693b42f4c97a03bb8de1a551df7 in term 0.
I20260924 19:14:47.096516 23072 leader_election.cc:302] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d06693b42f4c97a03bb8de1a551df7, d48f6cf8da014fb7aa70abce3b592bb9; no voters:
I20260924 19:14:47.096992 23208 raft_consensus.cc:2781] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:14:47.097164 23208 raft_consensus.cc:482] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:14:47.097287 23208 raft_consensus.cc:3037] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:14:47.099699 23208 raft_consensus.cc:504] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:47.100404 23208 leader_election.cc:288] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [CANDIDATE]: Term 1 election: Requested vote from peers 9460271a594e47168138112388d1aeaa (127.11.18.65:44643), d48f6cf8da014fb7aa70abce3b592bb9 (127.11.18.66:40351)
I20260924 19:14:47.100955 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf128a49298e4b0baa107a6770985c0d" candidate_uuid: "44d06693b42f4c97a03bb8de1a551df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9460271a594e47168138112388d1aeaa"
I20260924 19:14:47.101115 23003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf128a49298e4b0baa107a6770985c0d" candidate_uuid: "44d06693b42f4c97a03bb8de1a551df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48f6cf8da014fb7aa70abce3b592bb9"
I20260924 19:14:47.101188 22860 raft_consensus.cc:3037] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:14:47.101392 23003 raft_consensus.cc:3037] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:14:47.103710 22860 raft_consensus.cc:2454] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d06693b42f4c97a03bb8de1a551df7 in term 1.
I20260924 19:14:47.104045 23003 raft_consensus.cc:2454] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d06693b42f4c97a03bb8de1a551df7 in term 1.
I20260924 19:14:47.104249 23071 leader_election.cc:302] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d06693b42f4c97a03bb8de1a551df7, 9460271a594e47168138112388d1aeaa; no voters:
I20260924 19:14:47.104627 23208 raft_consensus.cc:2781] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:14:47.105050 23208 raft_consensus.cc:686] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 1 LEADER]: Becoming Leader. State: Replica: 44d06693b42f4c97a03bb8de1a551df7, State: Running, Role: LEADER
I20260924 19:14:47.105643 23208 consensus_queue.cc:237] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } }
I20260924 19:14:47.110889 22691 catalog_manager.cc:5495] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d06693b42f4c97a03bb8de1a551df7 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "44d06693b42f4c97a03bb8de1a551df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d06693b42f4c97a03bb8de1a551df7" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 } health_report { overall_health: UNKNOWN } } }
W20260924 19:14:47.155565 23049 tablet.cc:2369] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:14:47.187119 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:47.180226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.11.18.67:46837)
0924 19:14:47.180528 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:47.180535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:47.180683 (+ 148us) server_negotiation.cc:407] Connection header received
0924 19:14:47.181014 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:47.181019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:47.181103 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:47.181281 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:14:47.181736 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.182941 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.184539 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.184985 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.185141 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:47.185864 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:14:47.186424 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:14:47.186613 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:47.186722 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 19:14:47.186811 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260924 19:14:47.196436 23003 raft_consensus.cc:1261] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 1 FOLLOWER]: Refusing update from remote peer 44d06693b42f4c97a03bb8de1a551df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:14:47.196434 22860 raft_consensus.cc:1261] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 1 FOLLOWER]: Refusing update from remote peer 44d06693b42f4c97a03bb8de1a551df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:14:47.197621 23211 consensus_queue.cc:1035] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d48f6cf8da014fb7aa70abce3b592bb9" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:14:47.197988 23208 consensus_queue.cc:1035] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9460271a594e47168138112388d1aeaa" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:14:47.210978 23214 mvcc.cc:205] Tried to move back new op lower bound from 7332975768345403392 to 7332975767988703232. Current Snapshot: MvccSnapshot[applied={T|T < 7332975768345403392}]
I20260924 19:14:47.230113 22692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:14:47.230463 22692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:14:47.234817 22692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8369aa8461348bface45e48f2e11206: Sending DeleteTablet for 3 replicas of tablet bf128a49298e4b0baa107a6770985c0d
I20260924 19:14:47.236187 23117 tablet_service.cc:1514] Processing DeleteTablet for tablet bf128a49298e4b0baa107a6770985c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:14:47 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:51838
I20260924 19:14:47.236168 22983 tablet_service.cc:1514] Processing DeleteTablet for tablet bf128a49298e4b0baa107a6770985c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:14:47 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:41438
I20260924 19:14:47.237054 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet bf128a49298e4b0baa107a6770985c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:14:47 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:36496
I20260924 19:14:47.238004 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 22772
I20260924 19:14:47.249526 23227 tablet_replica.cc:330] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: stopping tablet replica
I20260924 19:14:47.253515 23227 raft_consensus.cc:2229] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:14:47.254419 23227 raft_consensus.cc:2258] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:14:47.256990 23227 ts_tablet_manager.cc:1905] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:14:47.269261 23227 ts_tablet_manager.cc:1918] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:14:47.269588 23227 log.cc:1196] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/bf128a49298e4b0baa107a6770985c0d
Sep 24 19:14:47 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:14:47.269960 23225 tablet_replica.cc:330] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: stopping tablet replica
I20260924 19:14:47.270411 23227 ts_tablet_manager.cc:1939] T bf128a49298e4b0baa107a6770985c0d P 9460271a594e47168138112388d1aeaa: Deleting consensus metadata
I20260924 19:14:47.270766 23225 raft_consensus.cc:2229] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:14:47.271618 23225 raft_consensus.cc:2258] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:14:47.272748 22677 catalog_manager.cc:4841] TS 9460271a594e47168138112388d1aeaa (127.11.18.65:44643): tablet bf128a49298e4b0baa107a6770985c0d (table test-table [id=90d078200024413ea1dc582c7a8a0305]) successfully deleted
I20260924 19:14:47.276813 23225 ts_tablet_manager.cc:1905] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:14:47.277876 23226 tablet_replica.cc:330] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: stopping tablet replica
I20260924 19:14:47.278441 23226 raft_consensus.cc:2229] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:14:47.278787 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:47.239645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.11.18.65:44643)
0924 19:14:47.239888 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:47.239896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:47.250490 (+ 10594us) server_negotiation.cc:407] Connection header received
0924 19:14:47.250784 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:47.250790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:47.250894 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:47.250978 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:14:47.257282 (+ 6304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.258446 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.264034 (+ 5588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.265137 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.265276 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:47.271248 (+ 5972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:14:47.271268 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:14:47.271284 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:14:47.271334 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:14:47.275099 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:47.276069 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:47.276079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:47.276082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:47.276262 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:47.276750 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:47.276755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:47.276758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:47.277217 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:14:47.277541 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:47.278143 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 19:14:47.278378 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:14:47.279371 23226 raft_consensus.cc:2258] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:14:47.280258 22900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:14:47.285130 23225 ts_tablet_manager.cc:1918] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:14:47.285384 23225 log.cc:1196] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/bf128a49298e4b0baa107a6770985c0d
I20260924 19:14:47.285478 23226 ts_tablet_manager.cc:1905] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:14:47.285967 23225 ts_tablet_manager.cc:1939] T bf128a49298e4b0baa107a6770985c0d P 44d06693b42f4c97a03bb8de1a551df7: Deleting consensus metadata
I20260924 19:14:47.287817 22676 catalog_manager.cc:4841] TS 44d06693b42f4c97a03bb8de1a551df7 (127.11.18.67:46837): tablet bf128a49298e4b0baa107a6770985c0d (table test-table [id=90d078200024413ea1dc582c7a8a0305]) successfully deleted
I20260924 19:14:47.292793 23226 ts_tablet_manager.cc:1918] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:14:47.293040 23226 log.cc:1196] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/bf128a49298e4b0baa107a6770985c0d
I20260924 19:14:47.293659 23226 ts_tablet_manager.cc:1939] T bf128a49298e4b0baa107a6770985c0d P d48f6cf8da014fb7aa70abce3b592bb9: Deleting consensus metadata
I20260924 19:14:47.295945 22678 catalog_manager.cc:4841] TS d48f6cf8da014fb7aa70abce3b592bb9 (127.11.18.66:40351): tablet bf128a49298e4b0baa107a6770985c0d (table test-table [id=90d078200024413ea1dc582c7a8a0305]) successfully deleted
I20260924 19:14:47.378099 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 22772
I20260924 19:14:47.398738 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 22911
Sep 24 19:14:47 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:14:47.417485 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:47.399697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.11.18.66:40351)
0924 19:14:47.399895 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:47.399903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:47.399927 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:14:47.400170 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:47.400175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:47.400245 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:47.400363 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:14:47.401974 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.402997 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.404446 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.404859 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.405006 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:47.410418 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:14:47.410439 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:14:47.410459 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:14:47.410513 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:14:47.413745 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:47.414854 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:47.414863 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:47.414866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:47.415015 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:47.415531 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:47.415535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:47.415538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:47.416072 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:14:47.416300 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:47.416858 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 19:14:47.417063 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:14:47.418339 23043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:14:47.515646 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 22911
I20260924 19:14:47.534052 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 23054
Sep 24 19:14:47 dist-test-slave-0lgr krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:14:47.554662 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:14:47.534892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.11.18.67:46837)
0924 19:14:47.535177 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 19:14:47.535183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:14:47.535206 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:14:47.535272 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:14:47.535278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:14:47.535366 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:14:47.535492 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:14:47.536849 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.538163 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.539773 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:14:47.540455 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:14:47.540622 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:14:47.545850 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:14:47.545867 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:14:47.545888 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:14:47.545955 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:14:47.550574 (+ 4619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:47.551811 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:47.551822 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:47.551827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:47.552050 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:14:47.552520 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:14:47.552526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:14:47.552531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:14:47.553024 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:14:47.553405 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0924 19:14:47.553937 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 19:14:47.554209 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260924 19:14:47.555488 23177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:14:47.685659 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 23054
I20260924 19:14:47.702651 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 22660
I20260924 19:14:47.704066 22722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:14:47.821162 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 22660
I20260924 19:14:47.851368 11337 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:14:48.187876 11337 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:14:48.188225 11337 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:14:48.534950 11337 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:14:48.535 UTC [22246] LOG: received smart shutdown request
2026-09-24 19:14:48.540 UTC [22246] LOG: background worker "logical replication launcher" (PID 22255) exited with exit code 1
2026-09-24 19:14:48.540 UTC [22250] LOG: shutting down
2026-09-24 19:14:48.807 UTC [22246] LOG: database system is shut down
2026-09-24T19:14:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54585 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:14:49 dist-test-slave-0lgr krb5kdc[23241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:14:49 dist-test-slave-0lgr krb5kdc[23241](info): set up 2 sockets
Sep 24 19:14:49 dist-test-slave-0lgr krb5kdc[23241](info): commencing operation
krb5kdc: starting...
W20260924 19:14:50.114540 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:14:50 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:14:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/rangeradmin_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/rangeradmin_127.11.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/rangerlookup_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/rangerlookup_127.11.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/HTTP_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/HTTP_127.11.18.84@KRBTEST.COM.keytab.
I20260924 19:14:50.486991 11337 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJqTBM_/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/postgres -l logfile start
2026-09-24 19:14:53.642 UTC [23271] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:14:53.642 UTC [23271] LOG: listening on IPv4 address "127.11.18.84", port 37253
2026-09-24 19:14:53.648 UTC [23271] LOG: listening on Unix socket "/tmp/.s.PGSQL.37253"
2026-09-24 19:14:53.676 UTC [23274] LOG: database system was shut down at 2026-09-24 19:14:51 UTC
2026-09-24 19:14:53.681 UTC [23271] LOG: database system is ready to accept connections
I20260924 19:14:54.717612 11337 mini_postgres.cc:96] Postgres bound to 37253
2026-09-24 19:14:54.735 UTC [23282] FATAL: database "slave" does not exist
127.11.18.84:37253 - accepting connections
I20260924 19:14:54.736615 11337 mini_ranger.cc:161] Starting Ranger...
I20260924 19:14:54.764771 11337 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:14:55.419251 11337 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:14:55.419355 11337 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-admin
2026-09-24 19:14:55,747 [I] DB FLAVOR :POSTGRES
2026-09-24 19:14:55,748 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:14:55,748 [I] Checking connection..
2026-09-24 19:14:55,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:14:56,014 [I] Checking connection passed.
2026-09-24 19:14:56,015 [I] --------- Verifying version history table ---------
2026-09-24 19:14:56,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:14:56,269 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:14:56,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:14:56,556 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:14:56,557 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:14:56,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:14:56.808 UTC [23346] WARNING: there is no transaction in progress
2026-09-24 19:14:56,816 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:14:56,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:14:57,077 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:14:57,077 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:14:57,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:14:57,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:14:57,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0lgr.c.gcp-upstream.internal','N') ;"
2026-09-24 19:14:57,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:14:58,123 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:14:58,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:14:58,412 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:14:58,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:14:58,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:14:58,953 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:14:58,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:15:00.149 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.171 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.194 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.214 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.238 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.272 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.358 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.369 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.390 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.403 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.418 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.438 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.450 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.460 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.471 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:00.480 UTC [23454] WARNING: there is no transaction in progress
2026-09-24 19:15:01,591 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:15:01,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:15:01,918 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:15:01,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:15:02,298 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:15:02,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:15:02,658 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:15:02,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:15:02,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:15:03,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0lgr.c.gcp-upstream.internal';"
2026-09-24 19:15:03,543 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:15:03,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:15:03,817 [I] DB_PATCHES have already been applied
I20260924 19:15:03.823712 11337 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJqTBM_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:15:03.823913 11337 mini_ranger.cc:193] Using host: 127.11.18.84
I20260924 19:15:03.838790 11337 mini_ranger.cc:227] Ranger admin URL: http://127.11.18.84:59593
Sep 24, 2026 7:15:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:15:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:15:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:15:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:15:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:15:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:15:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:15:05.058668 11337 mini_ranger.cc:160] Time spent starting Ranger: real 10.322s user 0.000s sys 0.040s
Sep 24, 2026 7:15:25 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:15:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 7:15:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:15:25 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:15:26 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:15:26.992383 11337 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/rangerkms_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/rangerkms_127.11.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.18.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.18.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.18.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/HTTP_127.11.18.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.18.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/HTTP_127.11.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:15:27 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277327, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:15:27.203878 11337 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:15:27.227555 11337 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:15:28.342690 11337 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:15:28.342814 11337 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms
I20260924 19:15:28.342854 11337 mini_ranger_kms.cc:227] Using postgres at 127.11.18.84:37253
2026-09-24 19:15:28,727 [I] DB FLAVOR :POSTGRES
2026-09-24 19:15:28,727 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:15:28,727 [I] Checking connection
2026-09-24 19:15:28,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:15:29,009 [I] connection success
2026-09-24 19:15:29,010 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:15:29,010 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:15:29,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:15:29,299 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:15:29,299 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:15:29,299 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:15:29,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.18.84:37253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:15:29,583 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:15:30.136897 11337 mini_ranger_kms.cc:310] Created kms service
I20260924 19:15:30.360256 11337 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:15:30.412441 11337 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:15:30 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277327, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:30.841821 11337 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:15:30.842195 11337 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJqTBM_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJqTBM_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:15:30.842259 11337 mini_ranger_kms.cc:240] Using host: 127.11.18.84
I20260924 19:15:30.863279 11337 mini_ranger_kms.cc:276] Ranger KMS PID: 23635
I20260924 19:15:30.863427 11337 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.18.84:46617
Sep 24, 2026 7:15:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:15:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:15:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:15:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:15:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:15:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:15:33.289081 11337 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.085s user 0.078s sys 0.330s
I20260924 19:15:33.289481 11337 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:15:33.289712 11337 mini_ranger_kms.cc:401] 127.11.18.84:46617/kms/v1/keys
Sep 24 19:15:33 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790277333, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.18.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:15:34 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277333, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.18.84@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
Sep 24 19:15:34 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277333, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.18.84@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
Sep 24 19:15:37 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:15:37.137996 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42921
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:42921
--ranger_config_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:46617/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:37.495405 23685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:37.495715 23685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:37.495862 23685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:37.504024 23685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:15:37.504169 23685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:37.504264 23685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:37.504354 23685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:15:37.504473 23685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:15:37.516426 23685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.18.84:46617/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:42921
--ranger_config_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:37.517927 23685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:37.519407 23685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:37.533977 23690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:37.534097 23693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:37.533990 23691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:37.534410 23685 server_base.cc:1034] running on GCE node
I20260924 19:15:37.535667 23685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:37.537376 23685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:37.538666 23685 hybrid_clock.cc:648] HybridClock initialized: now 1790277337538614 us; error 35 us; skew 500 ppm
I20260924 19:15:37.539088 23685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:37 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277337, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:37.542673 23685 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:15:37.544423 23685 webserver.cc:460] Webserver started at http://127.11.18.126:45777/ using document root <none> and password file <none>
I20260924 19:15:37.545125 23685 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:37.545274 23685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:37.545650 23685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:15:37 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277337, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:37.614388 23685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "31211127bbeb49bda10a39013efb726c"
format_stamp: "Formatted at 2026-09-24 19:15:37 on dist-test-slave-0lgr"
server_key: "733db3bd0d033c9563f24a75b0edc0b7"
server_key_iv: "3d353d8c11c8bb45094a37fa994d14a8"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:37.615347 23685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "31211127bbeb49bda10a39013efb726c"
format_stamp: "Formatted at 2026-09-24 19:15:37 on dist-test-slave-0lgr"
server_key: "733db3bd0d033c9563f24a75b0edc0b7"
server_key_iv: "3d353d8c11c8bb45094a37fa994d14a8"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:37.621256 23685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:15:37.645494 23700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:37.647917 23685 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.002s
I20260924 19:15:37.648144 23685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "31211127bbeb49bda10a39013efb726c"
format_stamp: "Formatted at 2026-09-24 19:15:37 on dist-test-slave-0lgr"
server_key: "733db3bd0d033c9563f24a75b0edc0b7"
server_key_iv: "3d353d8c11c8bb45094a37fa994d14a8"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:37.648420 23685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:37.686370 23685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:37.690701 23685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:37.690989 23685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:37.710465 23685 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:42921
I20260924 19:15:37.710503 23751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:42921 every 8 connection(s)
I20260924 19:15:37.711920 23685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:15:37.715970 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:37.717024 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 23685
I20260924 19:15:37.717252 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:15:37.717996 11337 external_mini_cluster.cc:1428] Setting key 59179997272916bf49d8605f9ac7ea9d
I20260924 19:15:37.731043 23752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Bootstrap starting.
Sep 24 19:15:37 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:37.734355 23752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:37.735772 23752 log.cc:824] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:37.739384 23752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: No bootstrap required, opened a new log
I20260924 19:15:37.744822 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:37.720075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.11.18.126:42921)
0924 19:15:37.720971 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:37.720987 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:37.721045 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:15:37.722976 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:37.723012 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:37.723605 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:37.724105 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:37.725590 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:37.727700 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:37.729098 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:37.730158 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:37.730347 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:37.734332 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:37.734346 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:37.734365 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:37.734416 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:37.738282 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:37.739026 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:37.739035 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:37.739043 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:37.739215 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:37.739813 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:37.739821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:37.739826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:37.740640 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:37.740940 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:37.742595 (+ 1655us) server_negotiation.cc:299] Negotiation successful
0924 19:15:37.743168 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":377,"threads_started":1}
I20260924 19:15:37.751016 23752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } }
I20260924 19:15:37.751415 23752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:37.751533 23752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31211127bbeb49bda10a39013efb726c, State: Initialized, Role: FOLLOWER
I20260924 19:15:37.752161 23752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } }
I20260924 19:15:37.752399 23752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:15:37.752534 23752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:15:37.752701 23752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:37.755160 23752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } }
I20260924 19:15:37.755740 23752 leader_election.cc:302] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31211127bbeb49bda10a39013efb726c; no voters:
I20260924 19:15:37.756469 23752 leader_election.cc:288] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:15:37.756593 23757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:15:37.757390 23757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 1 LEADER]: Becoming Leader. State: Replica: 31211127bbeb49bda10a39013efb726c, State: Running, Role: LEADER
I20260924 19:15:37.757966 23757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } }
I20260924 19:15:37.758744 23752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:15:37.760901 23758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31211127bbeb49bda10a39013efb726c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } } }
I20260924 19:15:37.761234 23758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:37.762152 23762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:15:37.761994 23759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31211127bbeb49bda10a39013efb726c. Latest consensus state: current_term: 1 leader_uuid: "31211127bbeb49bda10a39013efb726c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } } }
I20260924 19:15:37.762316 23759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:37.767179 23762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:15:37.778393 23752 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:15:37.778475 23762 catalog_manager.cc:1347] Generated new cluster ID: 0882f49d57de4384bf592f051f6d99f7
I20260924 19:15:37.778580 23762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:15:37.798487 23762 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:15:37.800631 23762 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:15:37.814097 23762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Generated new TSK 0
I20260924 19:15:37.815423 23762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:15:38.617595 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42921
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:15:38 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790277338, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:15:39.086112 23797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:39.086557 23797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:39.086726 23797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:39.098614 23797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:39.098773 23797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:39.099030 23797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:15:39.115339 23797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.18.84:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:39.117241 23797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:39.119050 23797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:39.140856 23808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:39.140869 23807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:39.141562 23810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:39.142890 23797 server_base.cc:1034] running on GCE node
I20260924 19:15:39.143500 23797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:39.148706 23797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:39.149981 23797 hybrid_clock.cc:648] HybridClock initialized: now 1790277339149916 us; error 63 us; skew 500 ppm
I20260924 19:15:39.150389 23797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:39 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277339, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:39.154821 23797 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:15:39.158749 23797 webserver.cc:460] Webserver started at http://127.11.18.65:45095/ using document root <none> and password file <none>
I20260924 19:15:39.159564 23797 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:39.159699 23797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:39.160070 23797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:15:39 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277339, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:39.180291 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "57ad2e8e060e45579124086a709fb2e9"
format_stamp: "Formatted at 2026-09-24 19:15:39 on dist-test-slave-0lgr"
server_key: "ffc8fd8e9253a16ad46986b003a907b3"
server_key_iv: "67f196181602727c7a40ab67c7a47be1"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:39.185683 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "57ad2e8e060e45579124086a709fb2e9"
format_stamp: "Formatted at 2026-09-24 19:15:39 on dist-test-slave-0lgr"
server_key: "ffc8fd8e9253a16ad46986b003a907b3"
server_key_iv: "67f196181602727c7a40ab67c7a47be1"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:39.204967 23797 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.012s sys 0.004s
I20260924 19:15:39.238955 23817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:39.240760 23797 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.009s
I20260924 19:15:39.240993 23797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "57ad2e8e060e45579124086a709fb2e9"
format_stamp: "Formatted at 2026-09-24 19:15:39 on dist-test-slave-0lgr"
server_key: "ffc8fd8e9253a16ad46986b003a907b3"
server_key_iv: "67f196181602727c7a40ab67c7a47be1"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:39.241284 23797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:39.271081 23797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:39.275678 23797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:39.276003 23797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:39.277100 23797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:39.278657 23797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:39.278762 23797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:39.278891 23797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:39.278980 23797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:39.338116 23797 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:43263
I20260924 19:15:39.338140 23929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:43263 every 8 connection(s)
I20260924 19:15:39.339984 23797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:15:39.346097 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 23797
I20260924 19:15:39.346318 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:15:39.347129 11337 external_mini_cluster.cc:1428] Setting key d5e2d7a4b8798b40fe43ac9a29832d99
Sep 24 19:15:39 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277339, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:15:39.385114 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:39.343001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:39137 (local address 127.11.18.126:42921)
0924 19:15:39.354121 (+ 11120us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:39.354131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:39.355705 (+ 1574us) server_negotiation.cc:407] Connection header received
0924 19:15:39.359194 (+ 3489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:39.359202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:39.359313 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:39.359436 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:39.365148 (+ 5712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:39.366229 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:39.369126 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:39.369844 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:39.369985 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:39.375192 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:39.375200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:39.375206 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:39.375251 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:39.380415 (+ 5164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:39.381466 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:39.381472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:39.381476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:39.381631 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:39.382213 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:39.382221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:39.382226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:39.382610 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:39.382755 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:39.384390 (+ 1635us) server_negotiation.cc:299] Negotiation successful
0924 19:15:39.384744 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10923,"thread_start_us":10788,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
I20260924 19:15:39.387956 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:39.343540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42921 (local address 127.11.18.65:39137)
0924 19:15:39.355445 (+ 11905us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:39.355518 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:39.358784 (+ 3266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:39.361858 (+ 3074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:39.361873 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:39.362616 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:39.363581 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:39.363609 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:39.366430 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:39.366436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:39.368574 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:39.368602 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:39.370337 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:39.370344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:39.370656 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:39.371551 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:39.371586 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:39.374624 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:39.380613 (+ 5989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:39.380624 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:39.380646 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:39.381285 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:39.381810 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:39.381815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:39.381818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:39.382073 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:39.382844 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:39.382853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:39.383384 (+ 531us) client_negotiation.cc:769] Sending connection context
0924 19:15:39.383691 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 19:15:39.386549 (+ 2858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11522,"thread_start_us":13770,"threads_started":1}
I20260924 19:15:39.390380 23930 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42921
I20260924 19:15:39.390709 23930 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:39.391592 23930 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
I20260924 19:15:39.394402 23717 ts_manager.cc:194] Registered new tserver with Master: 57ad2e8e060e45579124086a709fb2e9 (127.11.18.65:43263)
I20260924 19:15:39.396750 23717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:39137
Sep 24 19:15:39 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277338, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:15:39.441287 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42921
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:15:39 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277338, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
W20260924 19:15:39.912357 23936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:39.912914 23936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:39.913134 23936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:39.930583 23936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:39.930749 23936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:39.930999 23936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:15:39.948380 23936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.18.84:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:39.950506 23936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:39.952365 23936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:39.992333 23944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:39.993654 23943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:40.013634 23936 server_base.cc:1034] running on GCE node
W20260924 19:15:40.015020 23946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:40.015930 23936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:40.022838 23936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:40.024163 23936 hybrid_clock.cc:648] HybridClock initialized: now 1790277340024098 us; error 63 us; skew 500 ppm
I20260924 19:15:40.024547 23936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:40 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:40.028760 23936 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:15:40.034894 23936 webserver.cc:460] Webserver started at http://127.11.18.66:44883/ using document root <none> and password file <none>
I20260924 19:15:40.035809 23936 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:40.035952 23936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:40.036319 23936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:15:40 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:40.056087 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "0f3a41d86202482b9e0789ef948497b3"
format_stamp: "Formatted at 2026-09-24 19:15:40 on dist-test-slave-0lgr"
server_key: "f5c34050cb9b0fcfcd55dcbc2f1491ca"
server_key_iv: "c316af6af3796c8c36ac308c48dfcbf5"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:40.057685 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "0f3a41d86202482b9e0789ef948497b3"
format_stamp: "Formatted at 2026-09-24 19:15:40 on dist-test-slave-0lgr"
server_key: "f5c34050cb9b0fcfcd55dcbc2f1491ca"
server_key_iv: "c316af6af3796c8c36ac308c48dfcbf5"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:40.071146 23936 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.010s sys 0.001s
I20260924 19:15:40.103837 23955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:40.106204 23936 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.008s
I20260924 19:15:40.106446 23936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "0f3a41d86202482b9e0789ef948497b3"
format_stamp: "Formatted at 2026-09-24 19:15:40 on dist-test-slave-0lgr"
server_key: "f5c34050cb9b0fcfcd55dcbc2f1491ca"
server_key_iv: "c316af6af3796c8c36ac308c48dfcbf5"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:40.106798 23936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:40.146304 23936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:40.153703 23936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:40.154265 23936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:40.155692 23936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:40.157989 23936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:40.158098 23936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:40.158237 23936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:40.158349 23936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:40.203465 23936 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:40795
I20260924 19:15:40.203488 24072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:40795 every 8 connection(s)
I20260924 19:15:40.205420 23936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:15:40.210052 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 23936
I20260924 19:15:40.210275 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:15:40.210986 11337 external_mini_cluster.cc:1428] Setting key dfe96a7ae1b125e5e77ff696053ebbe0
Sep 24 19:15:40 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:15:40.259609 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:40.208241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:45169 (local address 127.11.18.126:42921)
0924 19:15:40.221715 (+ 13474us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:40.221724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:40.221748 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:15:40.221815 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:40.221824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:40.221948 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:40.222064 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:40.227453 (+ 5389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.228858 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.230873 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.231723 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.231897 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:40.237623 (+ 5726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:40.237635 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:40.237644 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:40.237698 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:40.242161 (+ 4463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:40.254128 (+ 11967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:40.254137 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:40.254141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:40.254383 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:40.255169 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:40.255175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:40.255179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:40.255731 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:40.255908 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:40.257578 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0924 19:15:40.257930 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13316,"thread_start_us":13187,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:15:40.260890 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:40.209159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42921 (local address 127.11.18.66:45169)
0924 19:15:40.218012 (+ 8853us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:40.218074 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:40.220932 (+ 2858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:40.225827 (+ 4895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:40.225838 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:40.226501 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:40.227211 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:40.227231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.229134 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.229138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:40.230688 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:40.230703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.232260 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.232270 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:40.232656 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:40.233573 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:40.233613 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:40.237184 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:40.242417 (+ 5233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:40.242433 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:40.242459 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:40.253841 (+ 11382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:40.254566 (+ 725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:40.254572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:40.254635 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:40.255009 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:40.255997 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:40.256007 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:40.256560 (+ 553us) client_negotiation.cc:769] Sending connection context
0924 19:15:40.256896 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 19:15:40.258680 (+ 1784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8466,"thread_start_us":8181,"threads_started":1}
I20260924 19:15:40.263054 24073 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42921
I20260924 19:15:40.263417 24073 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:40.264350 24073 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
I20260924 19:15:40.266311 23717 ts_manager.cc:194] Registered new tserver with Master: 0f3a41d86202482b9e0789ef948497b3 (127.11.18.66:40795)
I20260924 19:15:40.267861 23717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:45169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:15:40.298503 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42921
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:15:40.401153 23930 heartbeater.cc:503] Master 127.11.18.126:42921 was elected leader, sending a full tablet report...
W20260924 19:15:40.644421 24079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:40.644786 24079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:40.644963 24079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:40.652624 24079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:40.652794 24079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:40.652984 24079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:15:40.663507 24079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.18.84:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:40.664858 24079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:40.666184 24079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:40.679652 24085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:40.679697 24087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:40.679610 24084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:40.680719 24079 server_base.cc:1034] running on GCE node
I20260924 19:15:40.681546 24079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:40.682958 24079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:40.684351 24079 hybrid_clock.cc:648] HybridClock initialized: now 1790277340684291 us; error 50 us; skew 500 ppm
I20260924 19:15:40.684741 24079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:40 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:40.688069 24079 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:15:40.689670 24079 webserver.cc:460] Webserver started at http://127.11.18.67:46313/ using document root <none> and password file <none>
I20260924 19:15:40.690371 24079 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:40.690522 24079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:40.690831 24079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:15:40 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:40.710718 24079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "6d5028c4e9db442ea13bc81f74b4608f"
format_stamp: "Formatted at 2026-09-24 19:15:40 on dist-test-slave-0lgr"
server_key: "2c8a3ef4f7e6aac6fb90b7a2479bf2c8"
server_key_iv: "9a911d8407d2557d4170b060d8439a24"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:40.711833 24079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "6d5028c4e9db442ea13bc81f74b4608f"
format_stamp: "Formatted at 2026-09-24 19:15:40 on dist-test-slave-0lgr"
server_key: "2c8a3ef4f7e6aac6fb90b7a2479bf2c8"
server_key_iv: "9a911d8407d2557d4170b060d8439a24"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:40.719075 24079 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260924 19:15:40.735369 24094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:40.737156 24079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:15:40.737428 24079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "6d5028c4e9db442ea13bc81f74b4608f"
format_stamp: "Formatted at 2026-09-24 19:15:40 on dist-test-slave-0lgr"
server_key: "2c8a3ef4f7e6aac6fb90b7a2479bf2c8"
server_key_iv: "9a911d8407d2557d4170b060d8439a24"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:40.737707 24079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:40.755399 24079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:40.759850 24079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:40.760202 24079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:40.761207 24079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:40.762703 24079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:40.762826 24079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:40.762951 24079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:40.763000 24079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:40.797915 24079 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:38025
I20260924 19:15:40.798008 24206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:38025 every 8 connection(s)
I20260924 19:15:40.799216 24079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:15:40.804102 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 24079
I20260924 19:15:40.804301 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:15:40.804886 11337 external_mini_cluster.cc:1428] Setting key 06a014deddcc80ecd1ba9d886db1d8e2
Sep 24 19:15:40 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:40.820290 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:40.801901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:35251 (local address 127.11.18.126:42921)
0924 19:15:40.802431 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:40.802436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:40.803186 (+ 750us) server_negotiation.cc:407] Connection header received
0924 19:15:40.805629 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:40.805635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:40.805734 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:40.805864 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:40.807472 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.808326 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.809999 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.810431 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.810574 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:40.814075 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:40.814081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:40.814084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:40.814118 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:40.816743 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:40.817571 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:40.817576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:40.817578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:40.817699 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:40.818126 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:40.818130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:40.818132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:40.818563 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:40.818714 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:40.819649 (+ 935us) server_negotiation.cc:299] Negotiation successful
0924 19:15:40.819884 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":262,"threads_started":1}
I20260924 19:15:40.821208 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:40.802346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42921 (local address 127.11.18.67:35251)
0924 19:15:40.803022 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:40.803064 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:40.805269 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:40.806111 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:40.806120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:40.806688 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:40.807304 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:40.807321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.808476 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.808487 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:40.809796 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:40.809809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.810656 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.810663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:40.810902 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:40.811675 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:40.811699 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:40.813849 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:40.816884 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:40.816892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:40.816906 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:40.817442 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:40.817792 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:40.817795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:40.817797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:40.818022 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:40.818728 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:40.818737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:40.819175 (+ 438us) client_negotiation.cc:769] Sending connection context
0924 19:15:40.819526 (+ 351us) client_negotiation.cc:240] Negotiation successful
0924 19:15:40.820008 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":273,"threads_started":1}
I20260924 19:15:40.822731 24207 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42921
I20260924 19:15:40.823050 24207 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:40.823837 24207 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
I20260924 19:15:40.825256 23717 ts_manager.cc:194] Registered new tserver with Master: 6d5028c4e9db442ea13bc81f74b4608f (127.11.18.67:38025)
I20260924 19:15:40.826239 23717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:35251
I20260924 19:15:40.833067 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:15:40.834800 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 23797
Sep 24 19:15:40 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:15:40.853012 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:40.835798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.11.18.65:43263)
0924 19:15:40.836567 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:40.836574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:40.836601 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:15:40.836663 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:40.836670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:40.836893 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:40.837031 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:40.838649 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.839788 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.841394 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.841949 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.842055 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:40.846990 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:40.847000 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:40.847015 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:40.847053 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:40.850100 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:40.850882 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:40.850890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:40.850892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:40.851062 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:40.851483 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:40.851487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:40.851489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:40.851870 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:40.852131 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:40.852495 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 19:15:40.852713 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":353,"threads_started":1}
I20260924 19:15:40.854441 23925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:40.938572 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 23797
I20260924 19:15:40.955029 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 23936
Sep 24 19:15:40 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:15:40.972342 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:40.955993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.11.18.66:40795)
0924 19:15:40.956520 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:40.956527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:40.956552 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:15:40.956611 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:40.956617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:40.956890 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:40.957060 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:40.958509 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.959502 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.960841 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:40.961566 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:40.961667 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:40.966196 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:40.966211 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:40.966228 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:40.966268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:40.969657 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:40.970483 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:40.970491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:40.970494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:40.970644 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:40.971003 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:40.971006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:40.971008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:40.971345 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:40.971636 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:40.971851 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 19:15:40.972057 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260924 19:15:40.973652 24068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:41.057101 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 23936
I20260924 19:15:41.077389 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 24079
Sep 24 19:15:41 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:15:41.095834 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:41.078505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.11.18.67:38025)
0924 19:15:41.079206 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:41.079214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:41.079244 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:15:41.079397 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:41.079436 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:41.079710 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:41.079915 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:41.081592 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:41.082730 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:41.084164 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:41.084794 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:41.084953 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:41.089766 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:41.089780 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:41.089796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:41.089833 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:41.092840 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:41.093852 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:41.093859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:41.093862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:41.094023 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:41.094350 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:41.094353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:41.094355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:41.094709 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:41.094944 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:41.095234 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 19:15:41.095431 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":390,"threads_started":1}
I20260924 19:15:41.097234 24202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:41.182711 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 24079
I20260924 19:15:41.198299 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 23685
I20260924 19:15:41.199743 23747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:41.307070 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 23685
I20260924 19:15:41.332333 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42921
--webserver_interface=127.11.18.126
--webserver_port=45777
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:42921
--ranger_config_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:46617/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.18.84:46617/kms with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:41.670159 24219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:41.670608 24219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:41.670781 24219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:41.678354 24219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:15:41.678536 24219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:41.678635 24219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:41.678746 24219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:15:41.678817 24219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:15:41.690708 24219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.18.84:46617/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:42921
--ranger_config_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:41.692103 24219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:41.693544 24219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:41.707306 24225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:41.707290 24227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:41.707640 24224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:41.707792 24219 server_base.cc:1034] running on GCE node
I20260924 19:15:41.708815 24219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:41.710388 24219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:41.711621 24219 hybrid_clock.cc:648] HybridClock initialized: now 1790277341711572 us; error 51 us; skew 500 ppm
I20260924 19:15:41.711970 24219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:41 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277341, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:41.721956 24219 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:15:41.723702 24219 webserver.cc:460] Webserver started at http://127.11.18.126:45777/ using document root <none> and password file <none>
I20260924 19:15:41.724452 24219 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:41.724630 24219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:15:41 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277341, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:41.803197 24219 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 19:15:41.806644 24235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:41.808318 24219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:15:41.808532 24219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "31211127bbeb49bda10a39013efb726c"
format_stamp: "Formatted at 2026-09-24 19:15:37 on dist-test-slave-0lgr"
server_key: "733db3bd0d033c9563f24a75b0edc0b7"
server_key_iv: "3d353d8c11c8bb45094a37fa994d14a8"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:41.809667 24219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:41.834322 24219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:41.838683 24219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:41.839015 24219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:41.858762 24219 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:42921
I20260924 19:15:41.858784 24286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:42921 every 8 connection(s)
I20260924 19:15:41.860281 24219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:15:41.867794 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 24219
I20260924 19:15:41.867909 24287 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:15:41.868762 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:43263
--local_ip_for_outbound_sockets=127.11.18.65
--tserver_master_addrs=127.11.18.126:42921
--webserver_port=45095
--webserver_interface=127.11.18.65
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.18.84:46617/kms with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:15:41.871906 24287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Bootstrap starting.
I20260924 19:15:41.892845 24287 log.cc:824] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:41.902997 24287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:15:41.903596 24287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Bootstrap complete.
I20260924 19:15:41.914654 24287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } }
I20260924 19:15:41.915751 24287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31211127bbeb49bda10a39013efb726c, State: Initialized, Role: FOLLOWER
I20260924 19:15:41.916577 24287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } }
I20260924 19:15:41.916867 24287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:15:41.917022 24287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:15:41.917219 24287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:15:41.921723 24287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } }
I20260924 19:15:41.922333 24287 leader_election.cc:302] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31211127bbeb49bda10a39013efb726c; no voters:
I20260924 19:15:41.923098 24287 leader_election.cc:288] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:15:41.923275 24291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:15:41.924680 24291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [term 2 LEADER]: Becoming Leader. State: Replica: 31211127bbeb49bda10a39013efb726c, State: Running, Role: LEADER
I20260924 19:15:41.925417 24291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } }
I20260924 19:15:41.925866 24287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:15:41.928232 24293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31211127bbeb49bda10a39013efb726c. Latest consensus state: current_term: 2 leader_uuid: "31211127bbeb49bda10a39013efb726c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } } }
I20260924 19:15:41.928723 24293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:41.929395 24292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "31211127bbeb49bda10a39013efb726c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31211127bbeb49bda10a39013efb726c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42921 } } }
I20260924 19:15:41.929809 24292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:41.935134 24297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:15:41.943625 24297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:15:41.945829 24297 catalog_manager.cc:1259] Loaded cluster ID: 0882f49d57de4384bf592f051f6d99f7
I20260924 19:15:41.946085 24297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:15:41.951815 24297 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:15:41.954162 24297 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Loaded TSK: 0
I20260924 19:15:41.956540 24297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:15:41.960906 24287 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 19:15:42.352897 24289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:42.353341 24289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:42.353590 24289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:42.367748 24289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:42.367986 24289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:42.368203 24289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:15:42.384598 24289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.18.84:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:43263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=45095
--tserver_master_addrs=127.11.18.126:42921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:42.386405 24289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:42.388326 24289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:42.429401 24324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:42.429979 24325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:42.447161 24327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:42.447898 24289 server_base.cc:1034] running on GCE node
I20260924 19:15:42.448978 24289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:42.450429 24289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:42.451728 24289 hybrid_clock.cc:648] HybridClock initialized: now 1790277342451640 us; error 83 us; skew 500 ppm
I20260924 19:15:42.452212 24289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:42 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277342, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:42.456218 24289 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:15:42.470567 24289 webserver.cc:460] Webserver started at http://127.11.18.65:45095/ using document root <none> and password file <none>
I20260924 19:15:42.471428 24289 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:42.471797 24289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:15:42 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277342, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:42.531507 24289 fs_manager.cc:709] Time spent opening directory manager: real 0.034s user 0.019s sys 0.011s
I20260924 19:15:42.541615 24336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:42.543521 24289 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260924 19:15:42.543818 24289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "57ad2e8e060e45579124086a709fb2e9"
format_stamp: "Formatted at 2026-09-24 19:15:39 on dist-test-slave-0lgr"
server_key: "ffc8fd8e9253a16ad46986b003a907b3"
server_key_iv: "67f196181602727c7a40ab67c7a47be1"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:42.545452 24289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:42.590301 24289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:42.595120 24289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:42.595429 24289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:42.596539 24289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:42.597993 24289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:42.598122 24289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:42.598248 24289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:42.598333 24289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:42.664397 24289 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:43263
I20260924 19:15:42.664477 24448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:43263 every 8 connection(s)
I20260924 19:15:42.666716 24289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:15:42.673089 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 24289
I20260924 19:15:42.674311 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:40795
--local_ip_for_outbound_sockets=127.11.18.66
--tserver_master_addrs=127.11.18.126:42921
--webserver_port=44883
--webserver_interface=127.11.18.66
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.18.84:46617/kms with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:15:42 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277342, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:42.714561 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:42.670909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42921 (local address 127.11.18.65:54751)
0924 19:15:42.680814 (+ 9905us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:42.680897 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:42.684151 (+ 3254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:42.688120 (+ 3969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:42.688133 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:42.688851 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:42.689723 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:42.689753 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:42.692333 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:42.692342 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:42.694674 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:42.694695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:42.696295 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:42.696307 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:42.696789 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:42.701607 (+ 4818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:42.701642 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:42.704535 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:42.708892 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:42.708901 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:42.708914 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:42.709467 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:42.710117 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:42.710122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:42.710126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:42.710398 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:42.711737 (+ 1339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:42.711746 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:42.712169 (+ 423us) client_negotiation.cc:769] Sending connection context
0924 19:15:42.712496 (+ 327us) client_negotiation.cc:240] Negotiation successful
0924 19:15:42.713097 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9463,"thread_start_us":13385,"threads_started":1}
I20260924 19:15:42.714699 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:42.670860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:54751 (local address 127.11.18.126:42921)
0924 19:15:42.686417 (+ 15557us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:42.686439 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:42.686490 (+ 51us) server_negotiation.cc:407] Connection header received
0924 19:15:42.686907 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:42.686928 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:42.687445 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:42.687859 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:42.690032 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:42.692117 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:42.694899 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:42.695998 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:42.696200 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:42.704938 (+ 8738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:42.704965 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:42.704988 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:42.705041 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:42.708684 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:42.709682 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:42.709692 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:42.709701 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:42.709935 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:42.710592 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:42.710598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:42.710601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:42.711308 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:42.711642 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:42.712863 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0924 19:15:42.713397 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15257,"thread_start_us":15011,"threads_started":1}
I20260924 19:15:42.721133 24449 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42921
I20260924 19:15:42.721513 24449 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:42.722398 24449 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
W20260924 19:15:42.725925 24449 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:42921 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:15:42.726164 24449 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:42.726485 24449 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
I20260924 19:15:42.727998 24449 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:42.728367 24449 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
W20260924 19:15:42.729559 24449 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:15:43.144593 24453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:43.145102 24453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:43.145296 24453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:43.154054 24453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:43.154217 24453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:43.154388 24453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:15:43.164683 24453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.18.84:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:40795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=44883
--tserver_master_addrs=127.11.18.126:42921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:43.166136 24453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:43.167522 24453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:43.185806 24467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:43.186244 24468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:43.193169 24470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:43.194938 24453 server_base.cc:1034] running on GCE node
I20260924 19:15:43.195694 24453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:43.196995 24453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:43.198273 24453 hybrid_clock.cc:648] HybridClock initialized: now 1790277343198186 us; error 79 us; skew 500 ppm
I20260924 19:15:43.198724 24453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:43 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:43.203163 24453 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:15:43.214818 24453 webserver.cc:460] Webserver started at http://127.11.18.66:44883/ using document root <none> and password file <none>
I20260924 19:15:43.215839 24453 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:43.216084 24453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:15:43 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:43.247676 24453 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.007s
I20260924 19:15:43.251752 24480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:43.253739 24453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 19:15:43.253971 24453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "0f3a41d86202482b9e0789ef948497b3"
format_stamp: "Formatted at 2026-09-24 19:15:40 on dist-test-slave-0lgr"
server_key: "f5c34050cb9b0fcfcd55dcbc2f1491ca"
server_key_iv: "c316af6af3796c8c36ac308c48dfcbf5"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:43.255091 24453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:43.280207 24453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:43.284901 24453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:43.285244 24453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:43.286451 24453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:43.288484 24453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:43.288614 24453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:43.288767 24453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:43.288864 24453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:15:43 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:43.376360 24453 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:40795
I20260924 19:15:43.376451 24592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:40795 every 8 connection(s)
I20260924 19:15:43.378391 24453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:15:43.379974 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 24453
I20260924 19:15:43.380753 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.18.84:46617/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:38025
--local_ip_for_outbound_sockets=127.11.18.67
--tserver_master_addrs=127.11.18.126:42921
--webserver_port=46313
--webserver_interface=127.11.18.67
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.18.84:46617/kms with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:15:43 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:43.423069 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:43.381596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:54809 (local address 127.11.18.126:42921)
0924 19:15:43.399476 (+ 17880us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:43.399486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:43.399514 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:15:43.399592 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:43.399602 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:43.399713 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:43.399856 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:43.403578 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:43.404744 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:43.408024 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:43.408752 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:43.408901 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:43.414296 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:43.414307 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:43.414315 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:43.414373 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:43.418780 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:43.419923 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:43.419930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:43.419934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:43.420121 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:43.420712 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:43.420718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:43.420722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:43.421118 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:43.421259 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:43.422367 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0924 19:15:43.422635 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17694,"thread_start_us":17561,"threads_started":1}
I20260924 19:15:43.427173 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:43.382236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42921 (local address 127.11.18.66:54809)
0924 19:15:43.394083 (+ 11847us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:43.394154 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:43.396111 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:43.400270 (+ 4159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:43.400289 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:43.402263 (+ 1974us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:43.403268 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:43.403304 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:43.404987 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:43.404997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:43.407790 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:43.407815 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:43.409089 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:43.409101 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:43.409550 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:43.410484 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:43.410534 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:43.413860 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:43.419026 (+ 5166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:43.419038 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:43.419061 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:43.419757 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:43.420286 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:43.420292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:43.420296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:43.420574 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:43.421381 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:43.421391 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:43.421879 (+ 488us) client_negotiation.cc:769] Sending connection context
0924 19:15:43.423658 (+ 1779us) client_negotiation.cc:240] Negotiation successful
0924 19:15:43.425529 (+ 1871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11494,"thread_start_us":11245,"threads_started":1}
I20260924 19:15:43.429432 24593 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42921
I20260924 19:15:43.429800 24593 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:43.430711 24593 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
I20260924 19:15:43.433063 24252 ts_manager.cc:194] Registered new tserver with Master: 0f3a41d86202482b9e0789ef948497b3 (127.11.18.66:40795)
I20260924 19:15:43.435427 24252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:54809
I20260924 19:15:43.729709 24449 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:43.730142 24449 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
I20260924 19:15:43.731441 24252 ts_manager.cc:194] Registered new tserver with Master: 57ad2e8e060e45579124086a709fb2e9 (127.11.18.65:43263)
I20260924 19:15:43.732534 24252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:54751
Sep 24 19:15:43 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
W20260924 19:15:43.899732 24597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:43.900277 24597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:43.900480 24597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:43.912611 24597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:43.912834 24597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:43.913060 24597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:15:43.928807 24597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.18.84:46617/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:38025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=46313
--tserver_master_addrs=127.11.18.126:42921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:43.930685 24597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:43.932542 24597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:43.950388 24603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:43.951627 24604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:43.966686 24597 server_base.cc:1034] running on GCE node
W20260924 19:15:43.967571 24606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:43.968490 24597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:43.975692 24597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:43.976996 24597 hybrid_clock.cc:648] HybridClock initialized: now 1790277343976938 us; error 79 us; skew 500 ppm
I20260924 19:15:43.977528 24597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:43 dist-test-slave-0lgr krb5kdc[23241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:43.980814 24597 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:15:43.982770 24597 webserver.cc:460] Webserver started at http://127.11.18.67:46313/ using document root <none> and password file <none>
I20260924 19:15:43.983528 24597 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:43.983693 24597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:15:43 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:44.015038 24597 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.013s sys 0.000s
I20260924 19:15:44.020192 24614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:44.021908 24597 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260924 19:15:44.022109 24597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "6d5028c4e9db442ea13bc81f74b4608f"
format_stamp: "Formatted at 2026-09-24 19:15:40 on dist-test-slave-0lgr"
server_key: "2c8a3ef4f7e6aac6fb90b7a2479bf2c8"
server_key_iv: "9a911d8407d2557d4170b060d8439a24"
server_key_version: "kuduclusterkey@0"
I20260924 19:15:44.023185 24597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:44.065404 24597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:44.069809 24597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:44.070138 24597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:44.071277 24597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:44.072786 24597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:44.072912 24597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:44.073045 24597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:44.073136 24597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:15:44 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for HTTP/127.11.18.84@KRBTEST.COM
I20260924 19:15:44.163578 24597 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:38025
I20260924 19:15:44.163655 24727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:38025 every 8 connection(s)
I20260924 19:15:44.165442 24597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:15:44.166862 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 24597
Sep 24 19:15:44 dist-test-slave-0lgr krb5kdc[23241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:44.238479 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.167941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.11.18.126:42921)
0924 19:15:44.209820 (+ 41879us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.209831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.219144 (+ 9313us) server_negotiation.cc:407] Connection header received
0924 19:15:44.219467 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.219474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.219585 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.219662 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:44.222508 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.223691 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.225760 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.226441 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.226541 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.230023 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:44.230028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:44.230032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:44.230078 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:44.233949 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.235830 (+ 1881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.235836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.235839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.236025 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.236957 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.236964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.236968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.237239 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:44.237543 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.237859 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.238091 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41691,"thread_start_us":46176,"threads_started":1}
I20260924 19:15:44.241376 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.178404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:57133 (local address 127.11.18.126:42921)
0924 19:15:44.214046 (+ 35642us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.214055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.214085 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:15:44.214298 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.214308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.214460 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.214613 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:44.219245 (+ 4632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.220510 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.223263 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.223912 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.224006 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.231441 (+ 7435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:44.231448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:44.231453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:44.231519 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:44.235194 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.236633 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.236639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.236644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.236788 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.238894 (+ 2106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.238900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.238903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.239221 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:44.239365 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.240623 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.241051 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35482,"thread_start_us":31436,"threads_started":1}
I20260924 19:15:44.243259 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.177509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42921 (local address 127.11.18.67:57133)
0924 19:15:44.204907 (+ 27398us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:44.204987 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:44.208664 (+ 3677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:44.215350 (+ 6686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:44.215372 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:44.216371 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:44.217569 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.217608 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.220756 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.220762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.223082 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.223100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.224387 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.224396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.224715 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.225588 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:44.225621 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:44.231109 (+ 5488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:44.235916 (+ 4807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:44.235929 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:44.235950 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:44.236499 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:44.237608 (+ 1109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:44.237613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:44.237616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:44.238760 (+ 1144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:44.239474 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:44.239482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:44.239969 (+ 487us) client_negotiation.cc:769] Sending connection context
0924 19:15:44.241423 (+ 1454us) client_negotiation.cc:240] Negotiation successful
0924 19:15:44.242073 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26924,"thread_start_us":32260,"threads_started":1}
I20260924 19:15:44.245631 24728 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42921
I20260924 19:15:44.245990 24728 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:44.246937 24728 heartbeater.cc:511] Master 127.11.18.126:42921 requested a full tablet report, sending...
I20260924 19:15:44.250751 24252 ts_manager.cc:194] Registered new tserver with Master: 6d5028c4e9db442ea13bc81f74b4608f (127.11.18.67:38025)
I20260924 19:15:44.252625 24252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:57133
I20260924 19:15:44.257519 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:15:44.285557 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.264967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.11.18.126:42921)
0924 19:15:44.265162 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.265168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.266903 (+ 1735us) server_negotiation.cc:407] Connection header received
0924 19:15:44.267095 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.267101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.267192 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.267314 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:44.269483 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.270630 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.272839 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.273537 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.273686 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.278376 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:44.278383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:44.278389 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:44.278438 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:44.282077 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.282992 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.282998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.283001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.283167 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.283617 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.283622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.283626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.284008 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:44.284269 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.284897 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.285155 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:15:44.293089 24252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:15:44.304858 24252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:15:44.379887 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.349259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:40795 (local address 127.0.0.1:43180)
0924 19:15:44.364288 (+ 15029us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:44.364312 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:44.364437 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:44.368859 (+ 4422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:44.368863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:44.368888 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:44.369336 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.369350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.371713 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.371719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.374142 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.374158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.378706 (+ 4548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.378713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.378947 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.379010 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 19:15:44.379141 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 19:15:44.379361 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14874,"mutex_wait_us":1,"thread_start_us":13237,"threads_started":1}
I20260924 19:15:44.380486 24743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.349665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:38025 (local address 127.0.0.1:56952)
0924 19:15:44.369646 (+ 19981us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:44.369666 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:44.369782 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:44.370742 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:44.370747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:44.370772 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:44.371112 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.371125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.374043 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.374048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.376128 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.376144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.379665 (+ 3521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.379672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.379860 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.379897 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:15:44.379997 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:15:44.380136 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19851,"thread_start_us":18218,"threads_started":1}
I20260924 19:15:44.381408 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.349955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.11.18.67:38025)
0924 19:15:44.365570 (+ 15615us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.365584 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.369915 (+ 4331us) server_negotiation.cc:407] Connection header received
0924 19:15:44.370031 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.370037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.370253 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.370424 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:44.371311 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.373858 (+ 2547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.376342 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.379143 (+ 2801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.379282 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.379504 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.380858 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.381047 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15339,"thread_start_us":15632,"threads_started":1}
I20260924 19:15:44.387358 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.348666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43263 (local address 127.0.0.1:43444)
0924 19:15:44.363116 (+ 14450us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:44.363189 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:44.363488 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:44.368123 (+ 4635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:44.368135 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:44.368173 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:44.368685 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.368715 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.380664 (+ 11949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.380669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.382773 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.382790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.386562 (+ 3772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.386569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.386749 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.386783 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:15:44.386886 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:15:44.387040 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14034,"thread_start_us":20797,"threads_started":1}
I20260924 19:15:44.387364 24663 tablet_service.cc:1467] Processing CreateTablet for tablet 08908ef71533416ca262c459cebaa9f1 (DEFAULT_TABLE table=test-table [id=efaa10f214b74d299023eae26e84e5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:15:44.388729 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.349214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.11.18.65:43263)
0924 19:15:44.366382 (+ 17168us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.366391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.366426 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:15:44.366536 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.366544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.366835 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.368907 (+ 2072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:44.369309 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.375754 (+ 6445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.383023 (+ 7269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.384582 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.384731 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.384982 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.388215 (+ 3233us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.388395 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16990,"thread_start_us":16844,"threads_started":1}
I20260924 19:15:44.388988 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08908ef71533416ca262c459cebaa9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:44.394039 24384 tablet_service.cc:1467] Processing CreateTablet for tablet 08908ef71533416ca262c459cebaa9f1 (DEFAULT_TABLE table=test-table [id=efaa10f214b74d299023eae26e84e5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:15:44.395633 24384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08908ef71533416ca262c459cebaa9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:44.406383 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.349530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.11.18.66:40795)
0924 19:15:44.368005 (+ 18475us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.368016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.368046 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:15:44.368145 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.368152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.368391 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.368626 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:44.369578 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.371075 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.374573 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.376048 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.376185 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.376470 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.405762 (+ 29292us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.406012 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18338,"thread_start_us":18230,"threads_started":1}
I20260924 19:15:44.410804 24749 tablet_bootstrap.cc:492] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Bootstrap starting.
I20260924 19:15:44.411284 24528 tablet_service.cc:1467] Processing CreateTablet for tablet 08908ef71533416ca262c459cebaa9f1 (DEFAULT_TABLE table=test-table [id=efaa10f214b74d299023eae26e84e5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:15:44.412906 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08908ef71533416ca262c459cebaa9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:44.418092 24749 tablet_bootstrap.cc:654] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:44.419153 24749 log.cc:824] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:44.422588 24749 tablet_bootstrap.cc:492] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: No bootstrap required, opened a new log
I20260924 19:15:44.422907 24749 ts_tablet_manager.cc:1397] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 19:15:44.435343 24749 raft_consensus.cc:348] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.435845 24751 tablet_bootstrap.cc:492] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Bootstrap starting.
I20260924 19:15:44.435890 24749 raft_consensus.cc:374] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:44.435986 24749 raft_consensus.cc:729] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5028c4e9db442ea13bc81f74b4608f, State: Initialized, Role: FOLLOWER
I20260924 19:15:44.436815 24749 consensus_queue.cc:260] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.438030 24752 tablet_bootstrap.cc:492] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Bootstrap starting.
I20260924 19:15:44.439743 24751 tablet_bootstrap.cc:654] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:44.440456 24593 heartbeater.cc:503] Master 127.11.18.126:42921 was elected leader, sending a full tablet report...
I20260924 19:15:44.441116 24751 log.cc:824] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:44.441794 24752 tablet_bootstrap.cc:654] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:44.443069 24752 log.cc:824] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:44.448645 24728 heartbeater.cc:503] Master 127.11.18.126:42921 was elected leader, sending a full tablet report...
I20260924 19:15:44.449757 24749 ts_tablet_manager.cc:1428] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Time spent starting tablet: real 0.027s user 0.017s sys 0.000s
I20260924 19:15:44.468582 24752 tablet_bootstrap.cc:492] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: No bootstrap required, opened a new log
I20260924 19:15:44.468937 24752 ts_tablet_manager.cc:1397] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.008s
W20260924 19:15:44.469527 24731 tablet.cc:2369] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:15:44.476485 24751 tablet_bootstrap.cc:492] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: No bootstrap required, opened a new log
I20260924 19:15:44.476881 24751 ts_tablet_manager.cc:1397] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.018s
I20260924 19:15:44.481585 24752 raft_consensus.cc:348] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.482121 24752 raft_consensus.cc:374] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:44.482236 24752 raft_consensus.cc:729] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ad2e8e060e45579124086a709fb2e9, State: Initialized, Role: FOLLOWER
I20260924 19:15:44.483124 24752 consensus_queue.cc:260] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.491153 24751 raft_consensus.cc:348] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.491959 24751 raft_consensus.cc:374] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:44.492367 24449 heartbeater.cc:503] Master 127.11.18.126:42921 was elected leader, sending a full tablet report...
I20260924 19:15:44.493208 24752 ts_tablet_manager.cc:1428] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Time spent starting tablet: real 0.024s user 0.023s sys 0.001s
I20260924 19:15:44.493937 24751 raft_consensus.cc:729] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3a41d86202482b9e0789ef948497b3, State: Initialized, Role: FOLLOWER
I20260924 19:15:44.494900 24751 consensus_queue.cc:260] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.500017 24751 ts_tablet_manager.cc:1428] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Time spent starting tablet: real 0.023s user 0.008s sys 0.008s
I20260924 19:15:44.598285 24753 raft_consensus.cc:482] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:15:44.598618 24753 raft_consensus.cc:504] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.600381 24753 leader_election.cc:288] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ad2e8e060e45579124086a709fb2e9 (127.11.18.65:43263), 0f3a41d86202482b9e0789ef948497b3 (127.11.18.66:40795)
I20260924 19:15:44.607618 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.600851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:50527 (local address 127.11.18.65:43263)
0924 19:15:44.601075 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.601084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.601104 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:15:44.601186 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.601192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.601277 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.601451 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:44.602051 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.603093 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.605387 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.606665 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.606842 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.606926 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.607233 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.607370 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:15:44.607641 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.600551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43263 (local address 127.11.18.67:50527)
0924 19:15:44.600853 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:44.600881 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:44.601015 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:44.601618 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:44.601625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:44.601657 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:44.601880 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.601896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.603260 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.603265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.605161 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.605176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.606842 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.606848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.607042 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.607080 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:15:44.607185 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:15:44.607339 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260924 19:15:44.608299 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.601248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:39641 (local address 127.11.18.66:40795)
0924 19:15:44.601799 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.601805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.601826 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:15:44.601896 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.601901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.601984 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.602081 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:44.602849 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.603862 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.606211 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.607202 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.607340 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.607405 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.607932 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.608078 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376}
I20260924 19:15:44.608920 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.600801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:40795 (local address 127.11.18.67:39641)
0924 19:15:44.601483 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:44.601500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:44.601594 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:44.602240 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:44.602246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:44.602272 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:44.602665 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.602678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.604040 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.604045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.606031 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:44.606043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.607545 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.607550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:44.607738 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.607784 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:15:44.608516 (+ 732us) client_negotiation.cc:240] Negotiation successful
0924 19:15:44.608678 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":859,"threads_started":1}
I20260924 19:15:44.608897 24404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08908ef71533416ca262c459cebaa9f1" candidate_uuid: "6d5028c4e9db442ea13bc81f74b4608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ad2e8e060e45579124086a709fb2e9" is_pre_election: true
I20260924 19:15:44.609517 24404 raft_consensus.cc:2454] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d5028c4e9db442ea13bc81f74b4608f in term 0.
I20260924 19:15:44.609931 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08908ef71533416ca262c459cebaa9f1" candidate_uuid: "6d5028c4e9db442ea13bc81f74b4608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3a41d86202482b9e0789ef948497b3" is_pre_election: true
I20260924 19:15:44.610284 24616 leader_election.cc:302] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ad2e8e060e45579124086a709fb2e9, 6d5028c4e9db442ea13bc81f74b4608f; no voters:
I20260924 19:15:44.610494 24548 raft_consensus.cc:2454] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d5028c4e9db442ea13bc81f74b4608f in term 0.
I20260924 19:15:44.610721 24753 raft_consensus.cc:2781] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:15:44.610868 24753 raft_consensus.cc:482] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:15:44.610976 24753 raft_consensus.cc:3037] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:44.613451 24753 raft_consensus.cc:504] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.614158 24753 leader_election.cc:288] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [CANDIDATE]: Term 1 election: Requested vote from peers 57ad2e8e060e45579124086a709fb2e9 (127.11.18.65:43263), 0f3a41d86202482b9e0789ef948497b3 (127.11.18.66:40795)
I20260924 19:15:44.614627 24404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08908ef71533416ca262c459cebaa9f1" candidate_uuid: "6d5028c4e9db442ea13bc81f74b4608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ad2e8e060e45579124086a709fb2e9"
I20260924 19:15:44.614686 24548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08908ef71533416ca262c459cebaa9f1" candidate_uuid: "6d5028c4e9db442ea13bc81f74b4608f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3a41d86202482b9e0789ef948497b3"
I20260924 19:15:44.614867 24548 raft_consensus.cc:3037] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:44.614862 24404 raft_consensus.cc:3037] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:44.618197 24404 raft_consensus.cc:2454] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d5028c4e9db442ea13bc81f74b4608f in term 1.
I20260924 19:15:44.618196 24548 raft_consensus.cc:2454] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d5028c4e9db442ea13bc81f74b4608f in term 1.
I20260924 19:15:44.618649 24616 leader_election.cc:302] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ad2e8e060e45579124086a709fb2e9, 6d5028c4e9db442ea13bc81f74b4608f; no voters:
I20260924 19:15:44.618995 24753 raft_consensus.cc:2781] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:15:44.619453 24753 raft_consensus.cc:686] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 1 LEADER]: Becoming Leader. State: Replica: 6d5028c4e9db442ea13bc81f74b4608f, State: Running, Role: LEADER
I20260924 19:15:44.620016 24753 consensus_queue.cc:237] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } }
I20260924 19:15:44.625898 24252 catalog_manager.cc:5495] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d5028c4e9db442ea13bc81f74b4608f (127.11.18.67). New cstate: current_term: 1 leader_uuid: "6d5028c4e9db442ea13bc81f74b4608f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d5028c4e9db442ea13bc81f74b4608f" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 } health_report { overall_health: UNKNOWN } } }
W20260924 19:15:44.630981 24594 tablet.cc:2369] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:15:44.672202 24450 tablet.cc:2369] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:15:44.674369 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.668159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.11.18.67:38025)
0924 19:15:44.668363 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.668370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.668461 (+ 91us) server_negotiation.cc:407] Connection header received
0924 19:15:44.668736 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.668740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.668829 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.668990 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:15:44.669452 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.670177 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.671618 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.672116 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.672253 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.672987 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:15:44.673624 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:15:44.673789 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.673891 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.673976 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:15:44.682798 24548 raft_consensus.cc:1261] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 1 FOLLOWER]: Refusing update from remote peer 6d5028c4e9db442ea13bc81f74b4608f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:15:44.682946 24404 raft_consensus.cc:1261] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 1 FOLLOWER]: Refusing update from remote peer 6d5028c4e9db442ea13bc81f74b4608f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:15:44.683903 24766 consensus_queue.cc:1035] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ad2e8e060e45579124086a709fb2e9" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:15:44.684679 24753 consensus_queue.cc:1035] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3a41d86202482b9e0789ef948497b3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:15:44.699630 24769 mvcc.cc:205] Tried to move back new op lower bound from 7332976003810820096 to 7332976003569152000. Current Snapshot: MvccSnapshot[applied={T|T < 7332976003810820096}]
I20260924 19:15:44.725625 24252 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:15:44.725936 24252 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:15:44.730475 24252 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31211127bbeb49bda10a39013efb726c: Sending DeleteTablet for 3 replicas of tablet 08908ef71533416ca262c459cebaa9f1
I20260924 19:15:44.731781 24384 tablet_service.cc:1514] Processing DeleteTablet for tablet 08908ef71533416ca262c459cebaa9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:15:44 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:43444
I20260924 19:15:44.732729 24778 tablet_replica.cc:330] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: stopping tablet replica
I20260924 19:15:44.733043 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet 08908ef71533416ca262c459cebaa9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:15:44 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:43180
I20260924 19:15:44.733166 24778 raft_consensus.cc:2229] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:15:44.733706 24778 raft_consensus.cc:2258] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:15:44.734449 24663 tablet_service.cc:1514] Processing DeleteTablet for tablet 08908ef71533416ca262c459cebaa9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:15:44 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:56952
I20260924 19:15:44.736012 24778 ts_tablet_manager.cc:1905] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:15:44.739913 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 24289
I20260924 19:15:44.743130 24778 ts_tablet_manager.cc:1918] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:15:44.743304 24778 log.cc:1196] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/08908ef71533416ca262c459cebaa9f1
I20260924 19:15:44.743832 24778 ts_tablet_manager.cc:1939] T 08908ef71533416ca262c459cebaa9f1 P 57ad2e8e060e45579124086a709fb2e9: Deleting consensus metadata
I20260924 19:15:44.745757 24237 catalog_manager.cc:4841] TS 57ad2e8e060e45579124086a709fb2e9 (127.11.18.65:43263): tablet 08908ef71533416ca262c459cebaa9f1 (table test-table [id=efaa10f214b74d299023eae26e84e5f0]) successfully deleted
I20260924 19:15:44.767506 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.741249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.11.18.65:43263)
0924 19:15:44.741437 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.741443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.750194 (+ 8751us) server_negotiation.cc:407] Connection header received
0924 19:15:44.750414 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.750419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.750494 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.750577 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:44.752716 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.753828 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.755613 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.756272 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.756365 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.759959 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:44.759993 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:44.760013 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:44.760061 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:44.763993 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.764864 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.764873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.764889 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.765048 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.765543 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.765547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.765550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.766018 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:44.766285 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.766868 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.767115 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:15:44.768682 24780 tablet_replica.cc:330] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: stopping tablet replica
I20260924 19:15:44.769189 24444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:44.769853 24779 tablet_replica.cc:330] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: stopping tablet replica
I20260924 19:15:44.770257 24779 raft_consensus.cc:2229] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:15:44.770800 24779 raft_consensus.cc:2258] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:15:44.769287 24780 raft_consensus.cc:2229] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:15:44.771829 24780 raft_consensus.cc:2258] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:15:44.773890 24780 ts_tablet_manager.cc:1905] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:15:44.774163 24779 ts_tablet_manager.cc:1905] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:15:44.782150 24780 ts_tablet_manager.cc:1918] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:15:44.782352 24780 log.cc:1196] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/08908ef71533416ca262c459cebaa9f1
I20260924 19:15:44.782378 24779 ts_tablet_manager.cc:1918] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:15:44.782604 24779 log.cc:1196] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/08908ef71533416ca262c459cebaa9f1
I20260924 19:15:44.782913 24780 ts_tablet_manager.cc:1939] T 08908ef71533416ca262c459cebaa9f1 P 6d5028c4e9db442ea13bc81f74b4608f: Deleting consensus metadata
I20260924 19:15:44.783207 24779 ts_tablet_manager.cc:1939] T 08908ef71533416ca262c459cebaa9f1 P 0f3a41d86202482b9e0789ef948497b3: Deleting consensus metadata
I20260924 19:15:44.784828 24238 catalog_manager.cc:4841] TS 6d5028c4e9db442ea13bc81f74b4608f (127.11.18.67:38025): tablet 08908ef71533416ca262c459cebaa9f1 (table test-table [id=efaa10f214b74d299023eae26e84e5f0]) successfully deleted
I20260924 19:15:44.785571 24236 catalog_manager.cc:4841] TS 0f3a41d86202482b9e0789ef948497b3 (127.11.18.66:40795): tablet 08908ef71533416ca262c459cebaa9f1 (table test-table [id=efaa10f214b74d299023eae26e84e5f0]) successfully deleted
I20260924 19:15:44.937295 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 24289
I20260924 19:15:44.963351 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 24453
I20260924 19:15:44.982801 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:44.964623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.11.18.66:40795)
0924 19:15:44.964833 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:44.964839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:44.964862 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:15:44.964932 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:44.964938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:44.965036 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:44.965161 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:44.966988 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.968154 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.970144 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:44.970829 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:44.970958 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:44.974511 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:44.974533 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:44.974552 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:44.974600 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:44.979082 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.980210 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.980219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.980223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.980401 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:44.980962 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:44.980967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:44.980971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:44.981462 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:44.981752 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:44.982176 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 19:15:44.982411 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:15:44.983717 24588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:45.080879 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 24453
I20260924 19:15:45.097404 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 24597
I20260924 19:15:45.116209 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:45.098218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.11.18.67:38025)
0924 19:15:45.098417 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:45.098422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:45.098545 (+ 123us) server_negotiation.cc:407] Connection header received
0924 19:15:45.098744 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:45.098748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:45.098810 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:45.098893 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:45.100969 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:45.101971 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:45.104239 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:45.104739 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:45.104849 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:45.108727 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:45.108742 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:45.108761 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:45.108806 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:45.112141 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:45.113267 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:45.113291 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:45.113295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:45.113797 (+ 502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:45.114358 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:45.114370 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:45.114374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:45.114860 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:45.115104 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:45.115649 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 19:15:45.115848 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:15:45.116897 24723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:45.243355 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 24597
I20260924 19:15:45.263837 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 24219
I20260924 19:15:45.265069 24282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:45.383713 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 24219
I20260924 19:15:45.406168 11337 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:15:45.742293 11337 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:15:45.742523 11337 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:15:46.088729 11337 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:15:46.089 UTC [23271] LOG: received smart shutdown request
2026-09-24 19:15:46.093 UTC [23271] LOG: background worker "logical replication launcher" (PID 23280) exited with exit code 1
2026-09-24 19:15:46.093 UTC [23275] LOG: shutting down
2026-09-24 19:15:46.320 UTC [23271] LOG: database system is shut down
2026-09-24T19:15:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57520 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:15:46 dist-test-slave-0lgr krb5kdc[24794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:15:46 dist-test-slave-0lgr krb5kdc[24794](info): set up 2 sockets
Sep 24 19:15:46 dist-test-slave-0lgr krb5kdc[24794](info): commencing operation
krb5kdc: starting...
W20260924 19:15:47.618263 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:15:47 dist-test-slave-0lgr krb5kdc[24794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:15:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:15:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:15:47.866755 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:43189
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:43189,127.11.18.125:43997,127.11.18.124:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:48.206652 24807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:48.207041 24807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:48.207198 24807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:48.214674 24807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:15:48.214850 24807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:48.214936 24807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:48.215049 24807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:15:48.215139 24807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:15:48.225193 24807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:43189,127.11.18.125:43997,127.11.18.124:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:48.226667 24807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:48.228011 24807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:48.240664 24812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:48.240768 24815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:48.240839 24813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:48.242261 24807 server_base.cc:1034] running on GCE node
I20260924 19:15:48.243240 24807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:48.245571 24807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:48.246863 24807 hybrid_clock.cc:648] HybridClock initialized: now 1790277348246816 us; error 71 us; skew 500 ppm
I20260924 19:15:48.247380 24807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:48 dist-test-slave-0lgr krb5kdc[24794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:48.255452 24807 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:15:48.257180 24807 webserver.cc:460] Webserver started at http://127.11.18.126:42171/ using document root <none> and password file <none>
I20260924 19:15:48.258029 24807 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:48.258145 24807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:48.258427 24807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:15:48.261803 24807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "630d9d515d464b23b528039201bffaf0"
format_stamp: "Formatted at 2026-09-24 19:15:48 on dist-test-slave-0lgr"
server_key: "892fa035721cc0c9e145d42a39f31f8c"
server_key_iv: "6ede05687ad61ce99b60e5e763382666"
server_key_version: "encryptionkey@0"
I20260924 19:15:48.262635 24807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "630d9d515d464b23b528039201bffaf0"
format_stamp: "Formatted at 2026-09-24 19:15:48 on dist-test-slave-0lgr"
server_key: "892fa035721cc0c9e145d42a39f31f8c"
server_key_iv: "6ede05687ad61ce99b60e5e763382666"
server_key_version: "encryptionkey@0"
I20260924 19:15:48.268446 24807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:15:48.272239 24822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:48.273803 24807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:15:48.273974 24807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "630d9d515d464b23b528039201bffaf0"
format_stamp: "Formatted at 2026-09-24 19:15:48 on dist-test-slave-0lgr"
server_key: "892fa035721cc0c9e145d42a39f31f8c"
server_key_iv: "6ede05687ad61ce99b60e5e763382666"
server_key_version: "encryptionkey@0"
I20260924 19:15:48.274175 24807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:48.297088 24807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:48.302539 24807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:48.302820 24807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:48.324987 24807 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:43189
I20260924 19:15:48.325006 24873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:43189 every 8 connection(s)
I20260924 19:15:48.326550 24807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:15:48.330605 24874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:48.331018 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 24807
I20260924 19:15:48.331245 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:15:48.331903 11337 external_mini_cluster.cc:1428] Setting key a3058a1f5836eae3cb6ffe0013d935a6
I20260924 19:15:48.337634 24874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.18.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.125@KRBTEST.COM" created.
Sep 24 19:15:48 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:48.364326 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.342499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.11.18.126:43189)
0924 19:15:48.343272 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:48.343283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:48.343346 (+ 63us) server_negotiation.cc:407] Connection header received
0924 19:15:48.344714 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:48.344720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:48.344983 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:48.345218 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:48.346661 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.348004 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.349609 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.350218 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.350460 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:48.355809 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:48.355819 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:48.355832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:48.355867 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:48.359158 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:48.360014 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:48.360027 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:48.360029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:48.360277 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:48.360837 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:48.360846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:48.360852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:48.361759 (+ 907us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:48.361949 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:48.362703 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 19:15:48.363060 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":556,"thread_start_us":377,"threads_started":1}
I20260924 19:15:48.364384 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.341058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:43189 (local address 127.0.0.1:41366)
0924 19:15:48.341966 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.342011 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:48.344276 (+ 2265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:48.345229 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:48.345236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:48.346062 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:48.346440 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:48.346459 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.348174 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.348182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:48.349463 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:48.349476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.350487 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.350494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:48.350698 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:48.351477 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:48.351502 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:48.355526 (+ 4024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:48.359347 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:48.359361 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:48.359375 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:48.359860 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:48.360437 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:48.360441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:48.360444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:48.360691 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:48.361933 (+ 1242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:48.361946 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:48.362239 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 19:15:48.362561 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 19:15:48.363280 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":83,"thread_start_us":392,"threads_started":1}
I20260924 19:15:48.368438 24874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:15:48.369861 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.369218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39098)
0924 19:15:48.369494 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.369663 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 19:15:48.370218 24825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
I20260924 19:15:48.372273 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.371862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39104)
0924 19:15:48.372046 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.372126 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 19:15:48.372586 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:43997: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab.
Entry for principal kudu/127.11.18.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab.
I20260924 19:15:48.388937 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.125:43997
--webserver_interface=127.11.18.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab
--principal=kudu/127.11.18.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:43189,127.11.18.125:43997,127.11.18.124:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:15:48.421968 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } attempt: 1
I20260924 19:15:48.423270 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.422756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39112)
0924 19:15:48.422976 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.423120 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:15:48.424428 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.424051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39116)
0924 19:15:48.424207 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.424288 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:15:48.424635 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:43997: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
I20260924 19:15:48.492950 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } attempt: 2
I20260924 19:15:48.494406 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.493783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39122)
0924 19:15:48.494133 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.494231 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260924 19:15:48.495659 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.495235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39136)
0924 19:15:48.495429 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.495502 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 19:15:48.495874 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:43997: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
I20260924 19:15:48.587170 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } attempt: 3
I20260924 19:15:48.588419 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.587943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39148)
0924 19:15:48.588162 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.588277 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 19:15:48.589675 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.589247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39156)
0924 19:15:48.589451 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.589541 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 19:15:48.589905 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:43997: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
W20260924 19:15:48.733053 24881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260924 19:15:48.733165 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } attempt: 4
W20260924 19:15:48.733410 24881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:48.733589 24881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 19:15:48.734481 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.733953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39168)
0924 19:15:48.734255 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.734341 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260924 19:15:48.735590 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.735160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39170)
0924 19:15:48.735314 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.735402 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:15:48.735781 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:43997: Network error: Client connection negotiation failed: client connection to 127.11.18.125:43997: connect: Connection refused (error 111)
W20260924 19:15:48.741539 24881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:15:48.741688 24881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:48.741791 24881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:48.741887 24881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:15:48.741976 24881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:15:48.752288 24881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:43189,127.11.18.125:43997,127.11.18.124:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab
--principal=kudu/127.11.18.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.125:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.18.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:48.753791 24881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:48.755100 24881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:48.767179 24887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:48.767349 24886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:48.767704 24889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:48.768018 24881 server_base.cc:1034] running on GCE node
I20260924 19:15:48.768755 24881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:48.770263 24881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:48.771543 24881 hybrid_clock.cc:648] HybridClock initialized: now 1790277348771482 us; error 56 us; skew 500 ppm
I20260924 19:15:48.771909 24881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:48 dist-test-slave-0lgr krb5kdc[24794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:48.775298 24881 init.cc:377] Logged in from keytab as kudu/127.11.18.125@KRBTEST.COM (short username kudu)
I20260924 19:15:48.777136 24881 webserver.cc:460] Webserver started at http://127.11.18.125:39287/ using document root <none> and password file <none>
I20260924 19:15:48.778123 24881 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:48.778271 24881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:48.778606 24881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:15:48.781857 24881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/data/instance:
uuid: "3645f409590545f080f247762b65fe5e"
format_stamp: "Formatted at 2026-09-24 19:15:48 on dist-test-slave-0lgr"
server_key: "9c1c64c2096a5eb23e398ce932b8ed7d"
server_key_iv: "fcf4a0cc4547e1fa974c4676a8c6f120"
server_key_version: "encryptionkey@0"
I20260924 19:15:48.782704 24881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/wal/instance:
uuid: "3645f409590545f080f247762b65fe5e"
format_stamp: "Formatted at 2026-09-24 19:15:48 on dist-test-slave-0lgr"
server_key: "9c1c64c2096a5eb23e398ce932b8ed7d"
server_key_iv: "fcf4a0cc4547e1fa974c4676a8c6f120"
server_key_version: "encryptionkey@0"
I20260924 19:15:48.788108 24881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:15:48.792205 24896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:48.794067 24881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:15:48.794358 24881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/wal
uuid: "3645f409590545f080f247762b65fe5e"
format_stamp: "Formatted at 2026-09-24 19:15:48 on dist-test-slave-0lgr"
server_key: "9c1c64c2096a5eb23e398ce932b8ed7d"
server_key_iv: "fcf4a0cc4547e1fa974c4676a8c6f120"
server_key_version: "encryptionkey@0"
I20260924 19:15:48.794571 24881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:48.820779 24881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:48.825951 24881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:48.826306 24881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:48.846517 24881 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.125:43997
I20260924 19:15:48.846566 24947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.125:43997 every 8 connection(s)
I20260924 19:15:48.848049 24881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
I20260924 19:15:48.851976 24948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:48.852818 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 24881
I20260924 19:15:48.853029 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-1/wal/instance
I20260924 19:15:48.853648 11337 external_mini_cluster.cc:1428] Setting key b6364ee8234074981413a6c31892c757
I20260924 19:15:48.862710 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.18.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.124@KRBTEST.COM" created.
Sep 24 19:15:48 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:48.889361 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.865881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.11.18.126:43189)
0924 19:15:48.866464 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:48.866469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:48.867578 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 19:15:48.869822 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:48.869828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:48.869937 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:48.870066 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:48.871745 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.872497 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.874222 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.874689 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.874841 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:48.880870 (+ 6029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:48.880880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:48.880888 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:48.880949 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:48.885119 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:48.886210 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:48.886219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:48.886227 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:48.886441 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:48.887118 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:48.887122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:48.887124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:48.887520 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:48.887709 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:48.888727 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0924 19:15:48.889025 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":334,"threads_started":1}
I20260924 19:15:48.890750 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.866517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:43189 (local address 127.0.0.1:41368)
0924 19:15:48.867374 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.867427 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:48.869496 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:48.870332 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:48.870341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:48.870911 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:48.871534 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:48.871553 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.872692 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.872696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:48.874048 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:48.874061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.874952 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.874960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:48.875211 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:48.875955 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:48.875981 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:48.880542 (+ 4561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:48.885410 (+ 4868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:48.885422 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:48.885443 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:48.886023 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:48.886623 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:48.886634 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:48.886638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:48.886903 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:48.887695 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:48.887706 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:48.888271 (+ 565us) client_negotiation.cc:769] Sending connection context
0924 19:15:48.888584 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 19:15:48.889192 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":362,"threads_started":1}
I20260924 19:15:48.892848 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:15:48 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab.
Entry for principal kudu/127.11.18.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab.
I20260924 19:15:48.914050 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.893630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39180)
0924 19:15:48.893871 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.893889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:48.893988 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:48.895443 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:48.895448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:48.895948 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:48.896087 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:48.896096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.897586 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.897592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:48.898692 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:48.898705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.899687 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.899692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:48.900029 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:48.900729 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:48.900748 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:48.906606 (+ 5858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:48.911107 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:48.911112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:48.911115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:48.911719 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:48.912158 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:48.912162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:48.912164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:48.912296 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:48.913121 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:48.913129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:48.913275 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 19:15:48.913460 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 19:15:48.913697 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":19}
I20260924 19:15:48.914467 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.124:33165
--webserver_interface=127.11.18.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab
--principal=kudu/127.11.18.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:43189,127.11.18.125:43997,127.11.18.124:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:15:48.914592 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.894004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.11.18.125:43997)
0924 19:15:48.894720 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:48.894727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:48.894755 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:15:48.894802 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:48.894818 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:48.895024 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:48.895443 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:48.896265 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.897446 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.898842 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:48.899508 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:48.899673 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:48.906928 (+ 7255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:48.906946 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:48.906962 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:48.907005 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:48.910927 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:48.911871 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:48.911878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:48.911880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:48.912006 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:48.912466 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:48.912470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:48.912472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:48.912960 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:48.913176 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:48.913647 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 19:15:48.914015 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":558,"thread_start_us":422,"threads_started":1}
I20260924 19:15:48.917865 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:15:48.919260 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.918727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45848)
0924 19:15:48.918979 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.919095 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 19:15:48.919629 24897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
I20260924 19:15:48.921590 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.921076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45852)
0924 19:15:48.921245 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.921404 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 19:15:48.921914 24948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
I20260924 19:15:48.971328 24948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 1
I20260924 19:15:48.972648 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.972172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45864)
0924 19:15:48.972376 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.972492 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:15:48.973706 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:48.973381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45874)
0924 19:15:48.973539 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:48.973606 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 19:15:48.973992 24948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
I20260924 19:15:49.035069 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } attempt: 5
I20260924 19:15:49.042238 24948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 2
I20260924 19:15:49.043475 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.042890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45886)
0924 19:15:49.043171 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.043281 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:15:49.045291 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.044324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45892)
0924 19:15:49.045091 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.045161 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677}
W20260924 19:15:49.045571 24948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
I20260924 19:15:49.051159 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.035895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39194)
0924 19:15:49.036133 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.036159 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:49.036289 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:49.036666 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:49.036669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:49.037038 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:49.037223 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.037232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.038617 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.038622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.039603 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.039617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.040700 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.040705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.040888 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.041834 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:49.041867 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:49.044878 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:49.048696 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.048700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.048703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.049100 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.049553 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.049556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.049559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.049700 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.050377 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:49.050381 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:49.050528 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 19:15:49.050651 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 19:15:49.050915 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 19:15:49.051606 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.036061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.11.18.125:43997)
0924 19:15:49.036280 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.036286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.036310 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:15:49.036397 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.036402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.036480 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.036585 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.037415 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.038468 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.039764 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.040450 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.040602 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.045091 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.045099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.045103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.045142 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.048533 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.049231 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.049237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.049240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.049416 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.049821 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.049826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.049829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.050205 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.050343 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.050790 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.051093 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:15:49.052677 24874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:15:49.054049 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.053508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45908)
0924 19:15:49.053722 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.053847 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:15:49.055263 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.054847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45920)
0924 19:15:49.055033 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.055137 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 19:15:49.055541 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
I20260924 19:15:49.106792 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 1
I20260924 19:15:49.108168 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.107538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45932)
0924 19:15:49.107778 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.107915 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 19:15:49.109604 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.109071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45946)
0924 19:15:49.109273 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.109425 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 19:15:49.109895 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
I20260924 19:15:49.136833 24948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 3
I20260924 19:15:49.138304 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.137676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45952)
0924 19:15:49.138000 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.138127 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 19:15:49.139710 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.139099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45964)
0924 19:15:49.139471 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.139561 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
W20260924 19:15:49.140004 24948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
I20260924 19:15:49.178135 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 2
I20260924 19:15:49.179580 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.178913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45974)
0924 19:15:49.179230 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.179347 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 19:15:49.181018 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.180426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45984)
0924 19:15:49.180652 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.180766 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 19:15:49.181365 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
W20260924 19:15:49.262830 24956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:49.263136 24956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:49.263302 24956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:49.270691 24956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:15:49.270804 24956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:49.270851 24956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:49.270947 24956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:15:49.271039 24956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:15:49.282011 24956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:43189,127.11.18.125:43997,127.11.18.124:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab
--principal=kudu/127.11.18.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.124:33165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.18.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:49.283273 24948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 4
I20260924 19:15:49.283504 24956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:49.284476 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.283947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45986)
0924 19:15:49.284215 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.284313 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 19:15:49.285092 24956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:15:49.285496 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.285111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:45994)
0924 19:15:49.285257 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.285384 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:15:49.285701 24948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
I20260924 19:15:49.287616 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 3
I20260924 19:15:49.288688 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.288189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:46002)
0924 19:15:49.288403 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.288509 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 19:15:49.290184 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.289702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:46004)
0924 19:15:49.289954 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.290046 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260924 19:15:49.290395 24874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:33165: Network error: Client connection negotiation failed: client connection to 127.11.18.124:33165: connect: Connection refused (error 111)
W20260924 19:15:49.298280 24964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:49.298357 24961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:49.298370 24962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:49.299239 24956 server_base.cc:1034] running on GCE node
I20260924 19:15:49.300025 24956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:49.301676 24956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:49.302888 24956 hybrid_clock.cc:648] HybridClock initialized: now 1790277349302825 us; error 59 us; skew 500 ppm
I20260924 19:15:49.303229 24956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277349, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:49.306440 24956 init.cc:377] Logged in from keytab as kudu/127.11.18.124@KRBTEST.COM (short username kudu)
I20260924 19:15:49.308070 24956 webserver.cc:460] Webserver started at http://127.11.18.124:43551/ using document root <none> and password file <none>
I20260924 19:15:49.308741 24956 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:49.308844 24956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:49.309162 24956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:15:49.312275 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/data/instance:
uuid: "cca2b75e58674bdbbd0a5805f2f86f4e"
format_stamp: "Formatted at 2026-09-24 19:15:49 on dist-test-slave-0lgr"
server_key: "e3d2ba4d2cddd02863469dc5725ebf33"
server_key_iv: "558666d56be982e52de89b0087005f60"
server_key_version: "encryptionkey@0"
I20260924 19:15:49.313124 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/wal/instance:
uuid: "cca2b75e58674bdbbd0a5805f2f86f4e"
format_stamp: "Formatted at 2026-09-24 19:15:49 on dist-test-slave-0lgr"
server_key: "e3d2ba4d2cddd02863469dc5725ebf33"
server_key_iv: "558666d56be982e52de89b0087005f60"
server_key_version: "encryptionkey@0"
I20260924 19:15:49.318691 24956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:15:49.322443 24971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:49.324069 24956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:15:49.324267 24956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/wal
uuid: "cca2b75e58674bdbbd0a5805f2f86f4e"
format_stamp: "Formatted at 2026-09-24 19:15:49 on dist-test-slave-0lgr"
server_key: "e3d2ba4d2cddd02863469dc5725ebf33"
server_key_iv: "558666d56be982e52de89b0087005f60"
server_key_version: "encryptionkey@0"
I20260924 19:15:49.324527 24956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:49.349656 24956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:49.354085 24956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:49.354351 24956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:49.373857 24956 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.124:33165
I20260924 19:15:49.373908 25022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.124:33165 every 8 connection(s)
I20260924 19:15:49.375278 24956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
I20260924 19:15:49.379302 25023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:49.380611 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 24956
I20260924 19:15:49.380820 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277128821086-11337-0/minicluster-data/master-2/wal/instance
I20260924 19:15:49.381467 11337 external_mini_cluster.cc:1428] Setting key c9f8906706f7fa02496cb7ef58749519
I20260924 19:15:49.389207 25023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277349, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:49.408425 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.383428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.11.18.124:33165)
0924 19:15:49.384295 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.384316 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.384362 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:15:49.386055 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.386080 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.386536 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.386892 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.388816 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.390513 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.392316 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.393603 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.393789 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.398623 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.398632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.398647 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.398686 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.402624 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.403543 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.403553 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.403563 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.403746 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.404359 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.404365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.404370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.405271 (+ 901us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.405697 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.406387 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.406889 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":411,"threads_started":1}
I20260924 19:15:49.411702 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.392550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.11.18.126:43189)
0924 19:15:49.393027 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.393034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.393614 (+ 580us) server_negotiation.cc:407] Connection header received
0924 19:15:49.393911 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.393919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.394022 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.394141 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.395328 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.396515 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.398180 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.398723 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.398819 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.403160 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.403166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.403170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.403215 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.407670 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.408568 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.408573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.408577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.409617 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.410048 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.410053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.410056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.410342 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.410531 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.411016 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.411259 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":246,"threads_started":1}
I20260924 19:15:49.411706 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.392632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:43189 (local address 127.0.0.1:41372)
0924 19:15:49.393481 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.393516 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:49.393728 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:49.394208 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:49.394214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:49.394879 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:49.395158 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.395167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.396691 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.396695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.397981 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.397993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.398910 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.398916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.399199 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.399958 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:49.400012 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:49.402930 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:49.407843 (+ 4913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.407853 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.407858 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.408366 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.409762 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.409767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.409771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.409915 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.410505 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:49.410513 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:49.410708 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 19:15:49.410885 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 19:15:49.411259 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":442,"threads_started":1}
I20260924 19:15:49.413723 25023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277349, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:15:49.429575 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.411899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.11.18.126:43189)
0924 19:15:49.412215 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.412221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.412585 (+ 364us) server_negotiation.cc:407] Connection header received
0924 19:15:49.412648 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.412653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.412725 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.412798 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.414625 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.415649 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.417259 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.417803 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.417947 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.423193 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.423199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.423204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.423245 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.426218 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.427123 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.427130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.427133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.427315 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.427827 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.427836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.427842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.428216 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.428551 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.428855 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.429184 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":437,"threads_started":1}
I20260924 19:15:49.429636 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.414450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:43997 (local address 127.0.0.1:39210)
0924 19:15:49.414723 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.414742 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:49.414845 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:49.415294 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:49.415297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:49.415823 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:49.416029 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.416041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.417109 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.417115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.418475 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.418493 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.419537 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.419547 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.419804 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.420382 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:49.420409 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:49.423247 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:49.426650 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.426654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.426657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.427167 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.427606 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.427609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.427612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.427786 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.428322 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:49.428330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:49.428626 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 19:15:49.428801 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 19:15:49.429115 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 19:15:49.429672 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.414755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.11.18.125:43997)
0924 19:15:49.414968 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.414975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.414995 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:15:49.415054 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.415060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.415136 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.415246 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.416194 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.416946 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.418668 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.419270 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.419407 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.423455 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.423462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.423466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.423508 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.426481 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.427298 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.427302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.427305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.427478 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.427921 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.427925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.427927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.428191 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.428335 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.428912 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.429147 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:15:49.431233 25023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277349, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:15:49.445484 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.432305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.11.18.124:33165)
0924 19:15:49.432477 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.432482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.432501 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:15:49.432551 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.432558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.432627 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.432737 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.433521 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.434297 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.435742 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.436368 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.436458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.439949 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.439956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.439960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.439998 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.442948 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.443571 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.443578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.443582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.443694 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.444034 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.444041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.444045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.444330 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.444498 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.444828 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.445056 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:15:49.445608 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.432080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:46018)
0924 19:15:49.432254 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.432272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:49.432357 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:49.432790 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:49.432793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:49.433149 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:49.433301 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.433312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.434445 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.434451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.435589 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.435604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.436540 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.436548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.436759 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.437552 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:49.437570 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:49.439753 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:49.443098 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.443103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.443107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.443457 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.443802 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.443808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.443811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.443935 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.444473 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:49.444477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:49.444620 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 19:15:49.444758 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 19:15:49.445226 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":136}
I20260924 19:15:49.447644 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.431643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.11.18.125:43997)
0924 19:15:49.431796 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.431800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.431870 (+ 70us) server_negotiation.cc:407] Connection header received
0924 19:15:49.432041 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.432044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.432096 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.432208 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.433774 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.434746 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.436143 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.436863 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.437021 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.441664 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.441670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.441673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.441708 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.444543 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.445309 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.445317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.445320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.445533 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.446100 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.446115 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.446118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.446401 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.446734 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.447051 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.447301 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:15:49.450901 25023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: Bootstrap starting.
I20260924 19:15:49.453203 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:15:49.454396 25023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:49.455404 25023 log.cc:824] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:49.458806 25023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: No bootstrap required, opened a new log
I20260924 19:15:49.467600 24874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 4
I20260924 19:15:49.477257 25023 raft_consensus.cc:348] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.477950 25023 raft_consensus.cc:374] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:49.478140 25023 raft_consensus.cc:729] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca2b75e58674bdbbd0a5805f2f86f4e, State: Initialized, Role: FOLLOWER
I20260924 19:15:49.479266 25023 consensus_queue.cc:260] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.485417 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.459169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.11.18.124:33165)
0924 19:15:49.459356 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.459364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.462639 (+ 3275us) server_negotiation.cc:407] Connection header received
0924 19:15:49.462912 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.462917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.462989 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.463055 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.466309 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.467635 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.470547 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.471343 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.471467 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.476644 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.476650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.476658 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.476693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.481163 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.482484 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.482494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.482501 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.482691 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.483217 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.483222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.483226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.483647 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.483932 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.484694 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.484968 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:15:49.486388 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.462770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.11.18.125:43997)
0924 19:15:49.462985 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.462993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.464342 (+ 1349us) server_negotiation.cc:407] Connection header received
0924 19:15:49.464520 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.464526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.464611 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.464711 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.467071 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.468150 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.470953 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.471895 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.472021 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.476654 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.476660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.476664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.476707 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.480587 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.482378 (+ 1791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.482390 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.482397 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.482649 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.483390 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.483401 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.483408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.483954 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.484301 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.485222 (+ 921us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.485680 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:15:49.486600 25039 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.488700 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.458788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.11.18.126:43189)
0924 19:15:49.458990 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.458996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.462005 (+ 3009us) server_negotiation.cc:407] Connection header received
0924 19:15:49.462185 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.462195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.462304 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.462441 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.466312 (+ 3871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.467639 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.470588 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.471354 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.471456 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.478980 (+ 7524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.478989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.478997 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.479060 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.483828 (+ 4768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.484969 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.484975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.484977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.485141 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.485663 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.485667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.485671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.486066 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.486395 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.488215 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.488436 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:15:49.489410 25023 sys_catalog.cc:564] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:15:49.489586 25039 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
W20260924 19:15:49.504029 25050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:15:49.504240 25050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:15:49.505607 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.468207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:46036)
0924 19:15:49.468650 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.468671 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:49.468775 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:49.479943 (+ 11168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:49.479952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:49.480477 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:49.480727 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.480739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.490315 (+ 9576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.490321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.491777 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.491797 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.493943 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.493953 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.494251 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.495094 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:49.495112 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:49.497901 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:49.502397 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.502413 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.502418 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.502960 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.503458 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.503463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.503467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.503671 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.504521 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:49.504527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:49.504710 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 19:15:49.504865 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 19:15:49.505159 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315}
I20260924 19:15:49.506446 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.469000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.11.18.124:33165)
0924 19:15:49.476503 (+ 7503us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.476510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.476537 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:15:49.476597 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.476602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.476682 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.486847 (+ 10165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.487182 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.490170 (+ 2988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.491954 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.493108 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.493292 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.498123 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.498132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.498140 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.498187 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.502192 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.503138 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.503145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.503149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.503296 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.503803 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.503808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.503811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.504271 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.504441 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.505547 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.505999 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7337,"thread_start_us":8756,"threads_started":1}
I20260924 19:15:49.512740 24874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0: Bootstrap starting.
I20260924 19:15:49.503731 24956 master_runner.cc:186] Error getting master registration for 127.11.18.126:43189: OK, instance_id { permanent_uuid: "630d9d515d464b23b528039201bffaf0" instance_seqno: 1790277348318490 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:15:49.515791 24956 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:15:49.516741 24874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:49.518191 24874 log.cc:824] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:49.522085 24874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0: No bootstrap required, opened a new log
I20260924 19:15:49.530781 24874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.531175 24874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:49.531350 24874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630d9d515d464b23b528039201bffaf0, State: Initialized, Role: FOLLOWER
I20260924 19:15:49.531987 24874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.533026 25053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.533533 25053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:15:49.534653 24874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:15:49.547509 25064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:15:49.547653 25064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:15:49.550848 25053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:15:49.551151 25053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.552904 25053 leader_election.cc:288] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3645f409590545f080f247762b65fe5e (127.11.18.125:43997), cca2b75e58674bdbbd0a5805f2f86f4e (127.11.18.124:33165)
I20260924 19:15:49.554252 24998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "630d9d515d464b23b528039201bffaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" is_pre_election: true
I20260924 19:15:49.554889 24998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630d9d515d464b23b528039201bffaf0 in term 0.
I20260924 19:15:49.555583 24823 leader_election.cc:302] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630d9d515d464b23b528039201bffaf0, cca2b75e58674bdbbd0a5805f2f86f4e; no voters:
W20260924 19:15:49.556272 24828 tablet.cc:2369] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:15:49.557139 25053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:15:49.557301 25053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:15:49.557433 25053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:49.547091 24807 master_runner.cc:186] Error getting master registration for 127.11.18.125:43997: OK, instance_id { permanent_uuid: "3645f409590545f080f247762b65fe5e" instance_seqno: 1790277348840882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:15:49.560949 24807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:15:49.561161 25053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.562961 24998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "630d9d515d464b23b528039201bffaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e"
I20260924 19:15:49.563212 24998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:49.567010 24998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630d9d515d464b23b528039201bffaf0 in term 1.
I20260924 19:15:49.567550 24823 leader_election.cc:302] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630d9d515d464b23b528039201bffaf0, cca2b75e58674bdbbd0a5805f2f86f4e; no voters:
I20260924 19:15:49.570757 25053 leader_election.cc:288] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 3645f409590545f080f247762b65fe5e (127.11.18.125:43997), cca2b75e58674bdbbd0a5805f2f86f4e (127.11.18.124:33165)
I20260924 19:15:49.571009 25053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:15:49.571166 25053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 630d9d515d464b23b528039201bffaf0, State: Running, Role: LEADER
I20260924 19:15:49.571760 25053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.578217 25053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 630d9d515d464b23b528039201bffaf0. Latest consensus state: current_term: 1 leader_uuid: "630d9d515d464b23b528039201bffaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.578555 25053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:49.584018 25068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:15:49.584900 24948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } attempt: 5
I20260924 19:15:49.591727 25068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:15:49 dist-test-slave-0lgr krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277348, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
I20260924 19:15:49.603492 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.585542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:33165 (local address 127.0.0.1:46052)
0924 19:15:49.585788 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:49.585810 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:49.585951 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:49.586474 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:49.586479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:49.587055 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:49.587269 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.587282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.588766 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.588772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.590312 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:49.590328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.591721 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.591729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:49.592006 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.592738 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:49.592766 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:49.595624 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:49.600302 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.600309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.600314 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.600936 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.601460 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:49.601494 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:49.601498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:49.601670 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:49.602390 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:49.602396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:49.602609 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 19:15:49.602760 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 19:15:49.603072 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 19:15:49.604424 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:49.585639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.11.18.124:33165)
0924 19:15:49.585811 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:49.585817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:49.585879 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:15:49.586073 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:49.586077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:49.586186 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:49.586282 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:49.587470 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.588596 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.590537 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:49.591252 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:49.591398 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:49.595930 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:49.595937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:49.595942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:49.595986 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:49.600144 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.601085 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.601091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.601129 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.601310 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:49.601794 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:49.601799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:49.601803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:49.602232 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:49.602383 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:49.603809 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0924 19:15:49.604047 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 19:15:49.606500 24977 tablet.cc:2369] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:15:49.610304 24948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e: Bootstrap starting.
I20260924 19:15:49.613307 24998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [term 1 FOLLOWER]: Refusing update from remote peer 630d9d515d464b23b528039201bffaf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:15:49.614420 25053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:15:49.622510 24948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:49.626768 24948 log.cc:824] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:49.631785 24948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e: No bootstrap required, opened a new log
I20260924 19:15:49.640515 25039 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 630d9d515d464b23b528039201bffaf0. Latest consensus state: current_term: 1 leader_uuid: "630d9d515d464b23b528039201bffaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.640884 25039 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:15:49.643975 24948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.644511 24948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:49.644670 24948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3645f409590545f080f247762b65fe5e, State: Initialized, Role: FOLLOWER
I20260924 19:15:49.645521 24948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } }
I20260924 19:15:49.662176 25075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.663184 25066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "630d9d515d464b23b528039201bffaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.663760 25066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:49.664888 25068 catalog_manager.cc:1347] Generated new cluster ID: f335bd89204d4e768c29236b79252a5c
I20260924 19:15:49.664892 25075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:15:49.664999 25068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:15:49.667130 24948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:15:49.667182 24923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "630d9d515d464b23b528039201bffaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3645f409590545f080f247762b65fe5e" is_pre_election: true
I20260924 19:15:49.667912 24923 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630d9d515d464b23b528039201bffaf0 in term 0.
I20260924 19:15:49.670588 25039 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "630d9d515d464b23b528039201bffaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.670876 25039 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:15:49.671854 24921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:49.676380 24921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [term 1 FOLLOWER]: Refusing update from remote peer 630d9d515d464b23b528039201bffaf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:15:49.677531 25066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:15:49.678797 24922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "630d9d515d464b23b528039201bffaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3645f409590545f080f247762b65fe5e"
I20260924 19:15:49.692354 24922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630d9d515d464b23b528039201bffaf0 in term 1.
I20260924 19:15:49.720124 25068 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:15:49.722726 25068 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:15:49.745791 25068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0: Generated new TSK 0
I20260924 19:15:49.747787 25068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:15:49.751420 25075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 630d9d515d464b23b528039201bffaf0. Latest consensus state: current_term: 1 leader_uuid: "630d9d515d464b23b528039201bffaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.751781 25075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:15:49.752373 25087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "630d9d515d464b23b528039201bffaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.752803 25087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:15:49.754693 25053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "630d9d515d464b23b528039201bffaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "630d9d515d464b23b528039201bffaf0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 43189 } } peers { permanent_uuid: "3645f409590545f080f247762b65fe5e" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 43997 } } peers { permanent_uuid: "cca2b75e58674bdbbd0a5805f2f86f4e" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 33165 } } }
I20260924 19:15:49.754995 25053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630d9d515d464b23b528039201bffaf0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:49.793275 25090 catalog_manager.cc:1259] Loaded cluster ID: f335bd89204d4e768c29236b79252a5c
I20260924 19:15:49.793447 25090 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e: loading cluster ID for follower catalog manager: success
I20260924 19:15:49.798782 25090 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e: acquiring CA information for follower catalog manager: success
I20260924 19:15:49.809243 25090 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3645f409590545f080f247762b65fe5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:15:50.506412 25050 catalog_manager.cc:1259] Loaded cluster ID: f335bd89204d4e768c29236b79252a5c
I20260924 19:15:50.506525 25050 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: loading cluster ID for follower catalog manager: success
I20260924 19:15:50.509016 25050 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: acquiring CA information for follower catalog manager: success
I20260924 19:15:50.511008 25050 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cca2b75e58674bdbbd0a5805f2f86f4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:15:50.912957 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 24807
I20260924 19:15:50.914419 24869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:51.030449 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 24807
I20260924 19:15:51.052129 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 24881
I20260924 19:15:51.053478 24943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:51.163982 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 24881
I20260924 19:15:51.186956 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 24956
I20260924 19:15:51.188273 25018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:51.299013 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 24956
2026-09-24T19:15:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4839 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:15:51 dist-test-slave-0lgr krb5kdc[25102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:15:51 dist-test-slave-0lgr krb5kdc[25102](info): set up 2 sockets
Sep 24 19:15:51 dist-test-slave-0lgr krb5kdc[25102](info): commencing operation
krb5kdc: starting...
W20260924 19:15:52.518040 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:15:52 dist-test-slave-0lgr krb5kdc[25102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:15:52.766336 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:34029
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:34029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:53.102013 25115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:53.102391 25115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:53.102579 25115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:53.109989 25115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:15:53.110180 25115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:53.110266 25115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:53.110383 25115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:15:53.110481 25115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:15:53.121750 25115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:34029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:34029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:53.123173 25115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:53.124555 25115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:53.137538 25120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:53.137543 25121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:53.137579 25123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:53.137986 25115 server_base.cc:1034] running on GCE node
I20260924 19:15:53.139233 25115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:53.140801 25115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:53.142012 25115 hybrid_clock.cc:648] HybridClock initialized: now 1790277353141961 us; error 38 us; skew 500 ppm
I20260924 19:15:53.142429 25115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:53 dist-test-slave-0lgr krb5kdc[25102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277353, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:53.145843 25115 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:15:53.147505 25115 webserver.cc:460] Webserver started at http://127.11.18.126:35797/ using document root <none> and password file <none>
I20260924 19:15:53.148180 25115 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:53.148281 25115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:53.148537 25115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:15:53.151926 25115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "4c8e99765d884ccdacfc1c6465174170"
format_stamp: "Formatted at 2026-09-24 19:15:53 on dist-test-slave-0lgr"
server_key: "a258bfb0472655654e12b6d33471d73f"
server_key_iv: "53c31ee0068b81842cf89bc3479c2ae6"
server_key_version: "encryptionkey@0"
I20260924 19:15:53.152710 25115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "4c8e99765d884ccdacfc1c6465174170"
format_stamp: "Formatted at 2026-09-24 19:15:53 on dist-test-slave-0lgr"
server_key: "a258bfb0472655654e12b6d33471d73f"
server_key_iv: "53c31ee0068b81842cf89bc3479c2ae6"
server_key_version: "encryptionkey@0"
I20260924 19:15:53.158321 25115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260924 19:15:53.162027 25130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:53.163595 25115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:15:53.163763 25115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "4c8e99765d884ccdacfc1c6465174170"
format_stamp: "Formatted at 2026-09-24 19:15:53 on dist-test-slave-0lgr"
server_key: "a258bfb0472655654e12b6d33471d73f"
server_key_iv: "53c31ee0068b81842cf89bc3479c2ae6"
server_key_version: "encryptionkey@0"
I20260924 19:15:53.163945 25115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:53.190793 25115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:53.195559 25115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:53.195806 25115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:53.215371 25115 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:34029
I20260924 19:15:53.215401 25181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:34029 every 8 connection(s)
I20260924 19:15:53.216854 25115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:15:53.221066 25182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:53.222507 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 25115
I20260924 19:15:53.222740 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:15:53.223443 11337 external_mini_cluster.cc:1428] Setting key 8872959a6d0c7f4f64389cf91e5bfd15
I20260924 19:15:53.237185 25182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170: Bootstrap starting.
Sep 24 19:15:53 dist-test-slave-0lgr krb5kdc[25102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:53.241531 25182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:53.243124 25182 log.cc:824] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:53.247149 25182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170: No bootstrap required, opened a new log
I20260924 19:15:53.250335 25185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:53.225437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.11.18.126:34029)
0924 19:15:53.226280 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:53.226292 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:53.226347 (+ 55us) server_negotiation.cc:407] Connection header received
0924 19:15:53.227957 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:53.227984 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:53.228411 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:53.228758 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:53.230990 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:53.232355 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:53.234585 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:53.235465 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:53.235630 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:53.241308 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:53.241351 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:53.241381 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:53.241441 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:53.245671 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:53.246842 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:53.246848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:53.246854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:53.247028 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:53.247512 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:53.247516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:53.247518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:53.248077 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:53.248336 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:53.248876 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 19:15:53.249245 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":397,"threads_started":1}
I20260924 19:15:53.254858 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:15:53.257241 25182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8e99765d884ccdacfc1c6465174170" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34029 } }
I20260924 19:15:53.257786 25182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:53.257931 25182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8e99765d884ccdacfc1c6465174170, State: Initialized, Role: FOLLOWER
I20260924 19:15:53.258765 25182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8e99765d884ccdacfc1c6465174170" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34029 } }
I20260924 19:15:53.259060 25182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:15:53.259232 25182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:15:53.259423 25182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:53.261930 25182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8e99765d884ccdacfc1c6465174170" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34029 } }
I20260924 19:15:53.262498 25182 leader_election.cc:302] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c8e99765d884ccdacfc1c6465174170; no voters:
I20260924 19:15:53.263208 25182 leader_election.cc:288] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:15:53.263396 25187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:15:53.264179 25187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [term 1 LEADER]: Becoming Leader. State: Replica: 4c8e99765d884ccdacfc1c6465174170, State: Running, Role: LEADER
I20260924 19:15:53.264822 25187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8e99765d884ccdacfc1c6465174170" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34029 } }
I20260924 19:15:53.265627 25182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:15:53.267978 25188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c8e99765d884ccdacfc1c6465174170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8e99765d884ccdacfc1c6465174170" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34029 } } }
I20260924 19:15:53.268441 25188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:53.268673 25189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c8e99765d884ccdacfc1c6465174170. Latest consensus state: current_term: 1 leader_uuid: "4c8e99765d884ccdacfc1c6465174170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8e99765d884ccdacfc1c6465174170" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 34029 } } }
I20260924 19:15:53.269006 25189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170 [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:53.271139 25195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:15:53.275801 25195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:15:53.285100 25195 catalog_manager.cc:1347] Generated new cluster ID: c3489f5026dd495ba71d560ed20f38ab
I20260924 19:15:53.285233 25195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:15:53.312158 25195 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:15:53.313927 25195 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:15:53.326998 25195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c8e99765d884ccdacfc1c6465174170: Generated new TSK 0
I20260924 19:15:53.327916 25195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:15:53.369117 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 25115
I20260924 19:15:53.370579 25177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:53.471020 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 25115
2026-09-24T19:15:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2169 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:15:53 dist-test-slave-0lgr krb5kdc[25210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:15:53 dist-test-slave-0lgr krb5kdc[25210](info): set up 2 sockets
Sep 24 19:15:53 dist-test-slave-0lgr krb5kdc[25210](info): commencing operation
krb5kdc: starting...
W20260924 19:15:54.670612 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:15:54 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:15:54.918582 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42123
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:55.250880 25223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:55.251194 25223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:55.251394 25223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:55.258772 25223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:15:55.258888 25223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:55.258984 25223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:55.259073 25223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:15:55.259145 25223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:15:55.269147 25223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:55.270597 25223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:55.271921 25223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:55.284662 25231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:55.284653 25229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:55.284642 25228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:55.286038 25223 server_base.cc:1034] running on GCE node
I20260924 19:15:55.286883 25223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:55.288441 25223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:55.289646 25223 hybrid_clock.cc:648] HybridClock initialized: now 1790277355289599 us; error 34 us; skew 500 ppm
I20260924 19:15:55.290007 25223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:55 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277355, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:55.293308 25223 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:15:55.295043 25223 webserver.cc:460] Webserver started at http://127.11.18.126:34495/ using document root <none> and password file <none>
I20260924 19:15:55.295782 25223 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:55.295936 25223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:55.296260 25223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:15:55.299420 25223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "38e256758cfc4a9fb5b42d36b8b467dc"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "a01930c62c1da913ba8357c32d560d90"
server_key_iv: "a50df667e94d31f7b01b22045511e2eb"
server_key_version: "encryptionkey@0"
I20260924 19:15:55.300230 25223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "38e256758cfc4a9fb5b42d36b8b467dc"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "a01930c62c1da913ba8357c32d560d90"
server_key_iv: "a50df667e94d31f7b01b22045511e2eb"
server_key_version: "encryptionkey@0"
I20260924 19:15:55.306025 25223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:15:55.309965 25238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:55.311621 25223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:15:55.311834 25223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "38e256758cfc4a9fb5b42d36b8b467dc"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "a01930c62c1da913ba8357c32d560d90"
server_key_iv: "a50df667e94d31f7b01b22045511e2eb"
server_key_version: "encryptionkey@0"
I20260924 19:15:55.312067 25223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:55.340430 25223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:55.345525 25223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:55.346017 25223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:55.367738 25223 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:42123
I20260924 19:15:55.367791 25289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:42123 every 8 connection(s)
I20260924 19:15:55.369287 25223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:15:55.373129 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 25223
I20260924 19:15:55.373409 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:15:55.373826 25290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:55.374356 11337 external_mini_cluster.cc:1428] Setting key 8a331aec0637833990a97de9077c27ba
I20260924 19:15:55.390739 25290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Bootstrap starting.
Sep 24 19:15:55 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:55.395206 25290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:55.396772 25290 log.cc:824] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:55.400627 25290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: No bootstrap required, opened a new log
I20260924 19:15:55.402523 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:55.376727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.11.18.126:42123)
0924 19:15:55.377652 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:55.377666 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:55.377707 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:15:55.379234 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:55.379257 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:55.379655 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:55.379992 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:55.382156 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:55.383709 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:55.385768 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:55.386689 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:55.386866 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:55.392457 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:55.392471 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:55.392499 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:55.392569 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:55.397166 (+ 4597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:55.398236 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:55.398244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:55.398251 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:55.398605 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:55.399055 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:55.399060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:55.399065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:55.399825 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:55.400174 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:55.400517 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 19:15:55.401003 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":422,"threads_started":1}
I20260924 19:15:55.410215 25290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:15:55.410655 25290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:55.410740 25290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e256758cfc4a9fb5b42d36b8b467dc, State: Initialized, Role: FOLLOWER
I20260924 19:15:55.411290 25290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:15:55.411513 25290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:15:55.411609 25290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:15:55.411741 25290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:55.414343 25290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:15:55.414848 25290 leader_election.cc:302] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38e256758cfc4a9fb5b42d36b8b467dc; no voters:
I20260924 19:15:55.415508 25290 leader_election.cc:288] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:15:55.415723 25295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:15:55.416519 25295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 1 LEADER]: Becoming Leader. State: Replica: 38e256758cfc4a9fb5b42d36b8b467dc, State: Running, Role: LEADER
I20260924 19:15:55.417116 25295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:15:55.418023 25290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:15:55.420157 25296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } } }
I20260924 19:15:55.420660 25296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:55.421651 25297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38e256758cfc4a9fb5b42d36b8b467dc. Latest consensus state: current_term: 1 leader_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } } }
I20260924 19:15:55.422812 25297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:55.423527 25304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:15:55.428577 25304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:15:55.438877 25304 catalog_manager.cc:1347] Generated new cluster ID: 14f0383ddb43424dbb91f60e966aa0bd
I20260924 19:15:55.439000 25304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:15:55.465798 25304 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:15:55.467209 25304 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:15:55.476186 25304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Generated new TSK 0
I20260924 19:15:55.477149 25304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:15:55.537717 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42123
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:55.899205 25316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:55.899587 25316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:55.899690 25316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:55.907749 25316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:55.907891 25316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:55.908008 25316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:15:55.918537 25316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:55.920050 25316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:55.921415 25316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:55.936411 25321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:55.936472 25322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:55.936424 25324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:55.937215 25316 server_base.cc:1034] running on GCE node
I20260924 19:15:55.937956 25316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:55.938966 25316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:55.940205 25316 hybrid_clock.cc:648] HybridClock initialized: now 1790277355940161 us; error 32 us; skew 500 ppm
I20260924 19:15:55.940620 25316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:55 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277355, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:55.944111 25316 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:15:55.945945 25316 webserver.cc:460] Webserver started at http://127.11.18.65:42825/ using document root <none> and password file <none>
I20260924 19:15:55.946717 25316 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:55.946869 25316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:55.947201 25316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:15:55.950614 25316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "04c80d426b8d469cbf80e59ae5a054d6"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "e92471f99c002ed2c61d798577ee36a2"
server_key_iv: "ee0d692104a6308be9fc1b9dd52b858e"
server_key_version: "encryptionkey@0"
I20260924 19:15:55.951584 25316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "04c80d426b8d469cbf80e59ae5a054d6"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "e92471f99c002ed2c61d798577ee36a2"
server_key_iv: "ee0d692104a6308be9fc1b9dd52b858e"
server_key_version: "encryptionkey@0"
I20260924 19:15:55.957213 25316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:15:55.961265 25331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:55.963069 25316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 19:15:55.963263 25316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "04c80d426b8d469cbf80e59ae5a054d6"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "e92471f99c002ed2c61d798577ee36a2"
server_key_iv: "ee0d692104a6308be9fc1b9dd52b858e"
server_key_version: "encryptionkey@0"
I20260924 19:15:55.963481 25316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:55.980309 25316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:55.985195 25316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:55.985605 25316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:55.986660 25316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:55.988078 25316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:55.988153 25316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:55.988298 25316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:55.988368 25316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:56.027257 25316 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:38773
I20260924 19:15:56.027315 25443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:38773 every 8 connection(s)
I20260924 19:15:56.029039 25316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:15:56.033267 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 25316
I20260924 19:15:56.033521 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:15:56.034278 11337 external_mini_cluster.cc:1428] Setting key c30e5bd3b62a04f8ec3753af5dc41c88
Sep 24 19:15:56 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277355, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:56.054105 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:56.032050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:47339 (local address 127.11.18.126:42123)
0924 19:15:56.032548 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:56.032553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:56.033839 (+ 1286us) server_negotiation.cc:407] Connection header received
0924 19:15:56.036526 (+ 2687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:56.036533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:56.036636 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:56.036777 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:56.038630 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:56.039953 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:56.041921 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:56.042726 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:56.042889 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:56.047167 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:56.047173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:56.047177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:56.047216 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:56.050458 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:56.051385 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:56.051390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:56.051393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:56.051535 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:56.052049 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:56.052056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:56.052065 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:56.052366 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:56.052545 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:56.053384 (+ 839us) server_negotiation.cc:299] Negotiation successful
0924 19:15:56.053655 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":212,"threads_started":1}
I20260924 19:15:56.055279 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:56.032586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:47339)
0924 19:15:56.033660 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:56.033712 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:56.036116 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:56.037032 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:56.037041 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:56.037735 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:56.038421 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:56.038442 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:56.040158 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:56.040162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:56.041730 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:56.041745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:56.043209 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:56.043219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:56.043472 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:56.044311 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:56.044338 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:56.046860 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:56.050644 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:56.050656 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:56.050678 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:56.051215 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:56.051670 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:56.051676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:56.051680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:56.051907 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:56.052513 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:56.052523 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:56.052931 (+ 408us) client_negotiation.cc:769] Sending connection context
0924 19:15:56.053235 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 19:15:56.053895 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":378,"threads_started":1}
I20260924 19:15:56.057562 25444 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42123
I20260924 19:15:56.057924 25444 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:56.058779 25444 heartbeater.cc:511] Master 127.11.18.126:42123 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:15:56.061539 25255 ts_manager.cc:194] Registered new tserver with Master: 04c80d426b8d469cbf80e59ae5a054d6 (127.11.18.65:38773)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
I20260924 19:15:56.063405 25255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:47339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:15:56.098852 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42123
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:56.447094 25450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:56.447409 25450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:56.447571 25450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:56.455116 25450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:56.455241 25450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:56.455432 25450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:15:56.465782 25450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:56.467159 25450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:56.468612 25450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:56.482649 25455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:56.482810 25456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:56.483193 25458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:56.484210 25450 server_base.cc:1034] running on GCE node
I20260924 19:15:56.484972 25450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:56.486064 25450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:56.487331 25450 hybrid_clock.cc:648] HybridClock initialized: now 1790277356487243 us; error 75 us; skew 500 ppm
I20260924 19:15:56.487740 25450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:56 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277356, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:56.491592 25450 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:15:56.493255 25450 webserver.cc:460] Webserver started at http://127.11.18.66:42755/ using document root <none> and password file <none>
I20260924 19:15:56.494025 25450 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:56.494176 25450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:56.494484 25450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:15:56.497660 25450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "1f9a3b8824854431a8859c3b9d9814e3"
format_stamp: "Formatted at 2026-09-24 19:15:56 on dist-test-slave-0lgr"
server_key: "efb71a580c94e2aa7f8f8b812cb4d23a"
server_key_iv: "f790385836c387b198b3d2a80b2b786b"
server_key_version: "encryptionkey@0"
I20260924 19:15:56.498502 25450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "1f9a3b8824854431a8859c3b9d9814e3"
format_stamp: "Formatted at 2026-09-24 19:15:56 on dist-test-slave-0lgr"
server_key: "efb71a580c94e2aa7f8f8b812cb4d23a"
server_key_iv: "f790385836c387b198b3d2a80b2b786b"
server_key_version: "encryptionkey@0"
I20260924 19:15:56.504015 25450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:15:56.508105 25465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:56.509979 25450 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 19:15:56.510198 25450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "1f9a3b8824854431a8859c3b9d9814e3"
format_stamp: "Formatted at 2026-09-24 19:15:56 on dist-test-slave-0lgr"
server_key: "efb71a580c94e2aa7f8f8b812cb4d23a"
server_key_iv: "f790385836c387b198b3d2a80b2b786b"
server_key_version: "encryptionkey@0"
I20260924 19:15:56.510448 25450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:56.531136 25450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:56.536190 25450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:56.536671 25450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:56.537803 25450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:56.539340 25450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:56.539469 25450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:56.539642 25450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:56.539726 25450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:56.578758 25450 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:33769
I20260924 19:15:56.578824 25577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:33769 every 8 connection(s)
I20260924 19:15:56.580087 25450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:15:56.584120 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 25450
I20260924 19:15:56.584391 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:15:56.585041 11337 external_mini_cluster.cc:1428] Setting key c59d307226bec88055a5a1ab069ef810
Sep 24 19:15:56 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277356, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:56.603349 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:56.583224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:57885 (local address 127.11.18.126:42123)
0924 19:15:56.583775 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:56.583780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:56.585041 (+ 1261us) server_negotiation.cc:407] Connection header received
0924 19:15:56.587489 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:56.587495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:56.587583 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:56.587726 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:56.589359 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:56.590189 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:56.591884 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:56.592376 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:56.592527 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:56.596764 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:56.596770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:56.596773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:56.596810 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:56.599789 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:56.600673 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:56.600682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:56.600687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:56.600838 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:56.601317 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:56.601347 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:56.601353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:56.601662 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:56.601834 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:56.602605 (+ 771us) server_negotiation.cc:299] Negotiation successful
0924 19:15:56.602877 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":302,"threads_started":1}
I20260924 19:15:56.604525 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:56.583884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:57885)
0924 19:15:56.584856 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:56.584906 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:56.587198 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:56.587994 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:56.588004 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:56.588553 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:56.589147 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:56.589166 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:56.590360 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:56.590368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:56.591720 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:56.591733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:56.592652 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:56.592659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:56.592942 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:56.593782 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:56.593812 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:56.596473 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:56.599977 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:56.599988 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:56.600008 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:56.600529 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:56.600966 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:56.600972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:56.600976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:56.601193 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:56.601819 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:56.601828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:56.602227 (+ 399us) client_negotiation.cc:769] Sending connection context
0924 19:15:56.602530 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 19:15:56.603197 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":401,"threads_started":1}
I20260924 19:15:56.606357 25578 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42123
I20260924 19:15:56.606720 25578 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:15:56.607748 25578 heartbeater.cc:511] Master 127.11.18.126:42123 requested a full tablet report, sending...
I20260924 19:15:56.609794 25255 ts_manager.cc:194] Registered new tserver with Master: 1f9a3b8824854431a8859c3b9d9814e3 (127.11.18.66:33769)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
I20260924 19:15:56.611111 25255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:57885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:15:56.641553 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42123
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:56.986795 25584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:56.987109 25584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:56.987205 25584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:56.994623 25584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:56.994746 25584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:56.994863 25584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:15:57.005118 25584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:57.006633 25584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:57.007963 25584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:57.022549 25589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:57.022522 25592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:57.022725 25590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:57.023682 25584 server_base.cc:1034] running on GCE node
I20260924 19:15:57.024447 25584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:57.025559 25584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:57.026813 25584 hybrid_clock.cc:648] HybridClock initialized: now 1790277357026768 us; error 55 us; skew 500 ppm
I20260924 19:15:57.027204 25584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:57 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:57.031289 25584 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:15:57.033128 25584 webserver.cc:460] Webserver started at http://127.11.18.67:38787/ using document root <none> and password file <none>
I20260924 19:15:57.033955 25584 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:57.034121 25584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:57.034451 25584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:15:57.037736 25584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "d990c3e9c0ce4aec94ec2fac52a08112"
format_stamp: "Formatted at 2026-09-24 19:15:57 on dist-test-slave-0lgr"
server_key: "8a44059a36e2719a9886165890ebff41"
server_key_iv: "4f69affc7f6e909512171e5cbdec9758"
server_key_version: "encryptionkey@0"
I20260924 19:15:57.038643 25584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "d990c3e9c0ce4aec94ec2fac52a08112"
format_stamp: "Formatted at 2026-09-24 19:15:57 on dist-test-slave-0lgr"
server_key: "8a44059a36e2719a9886165890ebff41"
server_key_iv: "4f69affc7f6e909512171e5cbdec9758"
server_key_version: "encryptionkey@0"
I20260924 19:15:57.044791 25584 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 19:15:57.048920 25599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:57.050568 25584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:15:57.050781 25584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "d990c3e9c0ce4aec94ec2fac52a08112"
format_stamp: "Formatted at 2026-09-24 19:15:57 on dist-test-slave-0lgr"
server_key: "8a44059a36e2719a9886165890ebff41"
server_key_iv: "4f69affc7f6e909512171e5cbdec9758"
server_key_version: "encryptionkey@0"
I20260924 19:15:57.051018 25584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:57.067828 25444 heartbeater.cc:503] Master 127.11.18.126:42123 was elected leader, sending a full tablet report...
I20260924 19:15:57.085659 25584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:57.090541 25584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:57.090842 25584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:57.092041 25584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:57.093879 25584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:57.094024 25584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:57.094182 25584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:57.094256 25584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:57.130651 25584 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:39679
I20260924 19:15:57.130702 25711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:39679 every 8 connection(s)
I20260924 19:15:57.132107 25584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:15:57.136575 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 25584
I20260924 19:15:57.136785 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:15:57.137413 11337 external_mini_cluster.cc:1428] Setting key a06e2fb01cc85bb0b2ac3c72bac1d56b
Sep 24 19:15:57 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277357, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:57.156041 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.135173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:57179 (local address 127.11.18.126:42123)
0924 19:15:57.135713 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.135720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.137002 (+ 1282us) server_negotiation.cc:407] Connection header received
0924 19:15:57.139646 (+ 2644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.139652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.139773 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.139891 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:57.141874 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.142748 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.144471 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.144963 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.145086 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.149269 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:57.149275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:57.149278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:57.149316 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:57.152416 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.153263 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.153271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.153280 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.153484 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.153963 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.153969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.153972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.154340 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:57.154522 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.155351 (+ 829us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.155607 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":250,"threads_started":1}
I20260924 19:15:57.157081 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.135869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:57179)
0924 19:15:57.136793 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:57.136850 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:57.139169 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:57.140206 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:57.140216 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:57.140775 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:57.141678 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.141700 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.142942 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.142946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.144266 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.144279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.145229 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.145234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.145591 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.146371 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:57.146401 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:57.148977 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:57.152592 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:57.152602 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:57.152626 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:57.153112 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:57.153612 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:57.153617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:57.153621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:57.153834 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:57.154482 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:57.154493 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:57.154938 (+ 445us) client_negotiation.cc:769] Sending connection context
0924 19:15:57.155253 (+ 315us) client_negotiation.cc:240] Negotiation successful
0924 19:15:57.155888 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":367,"threads_started":1}
I20260924 19:15:57.158499 25712 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42123
I20260924 19:15:57.158841 25712 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:57.159621 25712 heartbeater.cc:511] Master 127.11.18.126:42123 requested a full tablet report, sending...
I20260924 19:15:57.161190 25255 ts_manager.cc:194] Registered new tserver with Master: d990c3e9c0ce4aec94ec2fac52a08112 (127.11.18.67:39679)
I20260924 19:15:57.162106 25255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:57179
I20260924 19:15:57.167114 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:15:57.185647 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.171887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.11.18.126:42123)
0924 19:15:57.172128 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.172138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.172530 (+ 392us) server_negotiation.cc:407] Connection header received
0924 19:15:57.172658 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.172663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.172737 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.172832 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:57.174371 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.175136 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.176549 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.177042 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.177177 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.179809 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:57.179814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:57.179818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:57.179854 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:57.182727 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.183505 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.183510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.183522 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.183654 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.184068 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.184071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.184073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.184453 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:57.184641 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.185066 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.185293 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:15:57.190488 25255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:15:57.198318 25255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:15:57.226440 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.214792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.11.18.65:38773)
0924 19:15:57.217561 (+ 2769us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.217570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.217606 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:15:57.217713 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.217720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.218000 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.218319 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:57.219998 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.221647 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.223959 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.225051 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.225211 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.225509 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.225887 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.226093 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2607,"thread_start_us":2419,"threads_started":1}
I20260924 19:15:57.226668 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.214486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:38773 (local address 127.0.0.1:34088)
0924 19:15:57.216771 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:57.216795 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:57.216947 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:57.218593 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:57.218599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:57.218626 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:57.219802 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.219815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.221859 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.221862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.223735 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.223747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.225346 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.225354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.225640 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.225686 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:15:57.225840 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 19:15:57.226083 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2134,"mutex_wait_us":63,"thread_start_us":1958,"threads_started":1}
I20260924 19:15:57.226758 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.212551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.11.18.66:33769)
0924 19:15:57.217895 (+ 5344us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.217902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.217927 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:15:57.218022 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.218029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.218287 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.218514 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:57.219549 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.220756 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.223417 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.225053 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.225983 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.226196 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.226274 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.226425 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5208,"thread_start_us":5122,"threads_started":1}
I20260924 19:15:57.227180 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.211806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:39679 (local address 127.0.0.1:40826)
0924 19:15:57.213857 (+ 2051us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:57.213909 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:57.214126 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:57.219117 (+ 4991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:57.219121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:57.219149 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:57.219712 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.219730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.221792 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.221798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.224059 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.224073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.225982 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.225988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.226144 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.226173 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 19:15:57.226264 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:15:57.226395 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1703,"mutex_wait_us":73,"thread_start_us":1529,"threads_started":1}
I20260924 19:15:57.227523 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.211838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:33769 (local address 127.0.0.1:47382)
0924 19:15:57.214785 (+ 2947us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:57.214804 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:57.214917 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:57.218568 (+ 3651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:57.218576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:57.218613 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:57.219068 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.219081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.220900 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.220908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.223224 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.223242 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.225367 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.225375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.225582 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.225680 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 19:15:57.225825 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 19:15:57.227054 (+ 1229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2819,"mutex_wait_us":1017,"thread_start_us":4241,"threads_started":1}
I20260924 19:15:57.228322 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.211906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40826 (local address 127.11.18.67:39679)
0924 19:15:57.218554 (+ 6648us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.218563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.218595 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:15:57.218679 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.218686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.218898 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.219131 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:57.220018 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.221647 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.224237 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.225754 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.227501 (+ 1747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.227717 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.227790 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.227965 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6480,"thread_start_us":6311,"threads_started":1}
I20260924 19:15:57.231143 25513 tablet_service.cc:1467] Processing CreateTablet for tablet ebf2840adb7b45fb9948440b7b01b2da (DEFAULT_TABLE table=test-table [id=70951eac45c040a68e8dd6e1f0caa959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:15:57.231146 25379 tablet_service.cc:1467] Processing CreateTablet for tablet ebf2840adb7b45fb9948440b7b01b2da (DEFAULT_TABLE table=test-table [id=70951eac45c040a68e8dd6e1f0caa959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:15:57.232702 25513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf2840adb7b45fb9948440b7b01b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:57.232702 25379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf2840adb7b45fb9948440b7b01b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:57.235100 25647 tablet_service.cc:1467] Processing CreateTablet for tablet ebf2840adb7b45fb9948440b7b01b2da (DEFAULT_TABLE table=test-table [id=70951eac45c040a68e8dd6e1f0caa959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:15:57.236564 25647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf2840adb7b45fb9948440b7b01b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:15:57.247725 25731 tablet_bootstrap.cc:492] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Bootstrap starting.
I20260924 19:15:57.248793 25732 tablet_bootstrap.cc:492] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Bootstrap starting.
I20260924 19:15:57.252451 25732 tablet_bootstrap.cc:654] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:57.252902 25731 tablet_bootstrap.cc:654] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:57.253887 25732 log.cc:824] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:57.254117 25731 log.cc:824] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:57.254199 25733 tablet_bootstrap.cc:492] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Bootstrap starting.
I20260924 19:15:57.257488 25733 tablet_bootstrap.cc:654] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Neither blocks nor log segments found. Creating new log.
I20260924 19:15:57.258540 25733 log.cc:824] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:57.261632 25733 tablet_bootstrap.cc:492] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: No bootstrap required, opened a new log
I20260924 19:15:57.261682 25732 tablet_bootstrap.cc:492] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: No bootstrap required, opened a new log
I20260924 19:15:57.261682 25731 tablet_bootstrap.cc:492] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: No bootstrap required, opened a new log
I20260924 19:15:57.261957 25732 ts_tablet_manager.cc:1397] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260924 19:15:57.261945 25733 ts_tablet_manager.cc:1397] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 19:15:57.261948 25731 ts_tablet_manager.cc:1397] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260924 19:15:57.270079 25733 raft_consensus.cc:348] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.270535 25733 raft_consensus.cc:374] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:57.270641 25733 raft_consensus.cc:729] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c80d426b8d469cbf80e59ae5a054d6, State: Initialized, Role: FOLLOWER
I20260924 19:15:57.271318 25733 consensus_queue.cc:260] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.273279 25733 ts_tablet_manager.cc:1428] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 19:15:57.273092 25732 raft_consensus.cc:348] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.273226 25731 raft_consensus.cc:348] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.273690 25732 raft_consensus.cc:374] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:57.273698 25731 raft_consensus.cc:374] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:15:57.273816 25731 raft_consensus.cc:729] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d990c3e9c0ce4aec94ec2fac52a08112, State: Initialized, Role: FOLLOWER
I20260924 19:15:57.273810 25732 raft_consensus.cc:729] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9a3b8824854431a8859c3b9d9814e3, State: Initialized, Role: FOLLOWER
I20260924 19:15:57.274479 25731 consensus_queue.cc:260] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.274480 25732 consensus_queue.cc:260] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.276615 25732 ts_tablet_manager.cc:1428] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 19:15:57.277024 25712 heartbeater.cc:503] Master 127.11.18.126:42123 was elected leader, sending a full tablet report...
I20260924 19:15:57.277493 25731 ts_tablet_manager.cc:1428] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260924 19:15:57.279331 25578 heartbeater.cc:503] Master 127.11.18.126:42123 was elected leader, sending a full tablet report...
W20260924 19:15:57.281661 25445 tablet.cc:2369] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:15:57.316062 25738 raft_consensus.cc:482] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:15:57.316485 25738 raft_consensus.cc:504] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.318398 25738 leader_election.cc:288] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9a3b8824854431a8859c3b9d9814e3 (127.11.18.66:33769), 04c80d426b8d469cbf80e59ae5a054d6 (127.11.18.65:38773)
I20260924 19:15:57.325722 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.318989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:60591 (local address 127.11.18.66:33769)
0924 19:15:57.319231 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.319238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.319450 (+ 212us) server_negotiation.cc:407] Connection header received
0924 19:15:57.319725 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.319733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.319813 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.319925 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:57.320410 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.321437 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.323479 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.324725 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.324841 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.324894 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.325430 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.325530 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:15:57.326051 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.319080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:33769 (local address 127.11.18.67:60591)
0924 19:15:57.319345 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:57.319373 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:57.319527 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:57.319948 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:57.319953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:57.319983 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:57.320246 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.320268 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.321580 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.321586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.323323 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.323337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.324893 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.324902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.325140 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.325182 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:15:57.325298 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:15:57.325501 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":3}
I20260924 19:15:57.327291 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.319065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:34557 (local address 127.11.18.65:38773)
0924 19:15:57.319509 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.319515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.320155 (+ 640us) server_negotiation.cc:407] Connection header received
0924 19:15:57.320275 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.320281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.320373 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.320465 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:15:57.321203 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.322351 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.324695 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.326214 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.326425 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.326519 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.326882 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.327025 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260924 19:15:57.327351 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.319112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:38773 (local address 127.11.18.67:34557)
0924 19:15:57.320047 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:57.320061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:57.320168 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:57.320489 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:57.320496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:57.320522 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:15:57.320863 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.320876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.322504 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.322508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.324497 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:57.324514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.326544 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.326554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:57.326702 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.326727 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:15:57.326854 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 19:15:57.327000 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":608,"threads_started":1}
I20260924 19:15:57.327488 25533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf2840adb7b45fb9948440b7b01b2da" candidate_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9a3b8824854431a8859c3b9d9814e3" is_pre_election: true
I20260924 19:15:57.328199 25533 raft_consensus.cc:2454] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d990c3e9c0ce4aec94ec2fac52a08112 in term 0.
I20260924 19:15:57.328440 25399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf2840adb7b45fb9948440b7b01b2da" candidate_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c80d426b8d469cbf80e59ae5a054d6" is_pre_election: true
I20260924 19:15:57.329069 25399 raft_consensus.cc:2454] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d990c3e9c0ce4aec94ec2fac52a08112 in term 0.
I20260924 19:15:57.329183 25602 leader_election.cc:302] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9a3b8824854431a8859c3b9d9814e3, d990c3e9c0ce4aec94ec2fac52a08112; no voters:
I20260924 19:15:57.329704 25738 raft_consensus.cc:2781] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:15:57.329962 25738 raft_consensus.cc:482] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:15:57.330143 25738 raft_consensus.cc:3037] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:15:57.332594 25579 tablet.cc:2369] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:15:57.332753 25738 raft_consensus.cc:504] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.333715 25738 leader_election.cc:288] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9a3b8824854431a8859c3b9d9814e3 (127.11.18.66:33769), 04c80d426b8d469cbf80e59ae5a054d6 (127.11.18.65:38773)
I20260924 19:15:57.334261 25533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf2840adb7b45fb9948440b7b01b2da" candidate_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9a3b8824854431a8859c3b9d9814e3"
I20260924 19:15:57.334272 25399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf2840adb7b45fb9948440b7b01b2da" candidate_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c80d426b8d469cbf80e59ae5a054d6"
I20260924 19:15:57.334483 25533 raft_consensus.cc:3037] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:57.334501 25399 raft_consensus.cc:3037] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:15:57.336989 25533 raft_consensus.cc:2454] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d990c3e9c0ce4aec94ec2fac52a08112 in term 1.
I20260924 19:15:57.337014 25399 raft_consensus.cc:2454] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d990c3e9c0ce4aec94ec2fac52a08112 in term 1.
I20260924 19:15:57.337589 25601 leader_election.cc:302] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c80d426b8d469cbf80e59ae5a054d6, d990c3e9c0ce4aec94ec2fac52a08112; no voters:
I20260924 19:15:57.338057 25738 raft_consensus.cc:2781] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:15:57.338599 25738 raft_consensus.cc:686] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 LEADER]: Becoming Leader. State: Replica: d990c3e9c0ce4aec94ec2fac52a08112, State: Running, Role: LEADER
I20260924 19:15:57.339125 25738 consensus_queue.cc:237] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } }
I20260924 19:15:57.344640 25255 catalog_manager.cc:5495] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 reported cstate change: term changed from 0 to 1, leader changed from <none> to d990c3e9c0ce4aec94ec2fac52a08112 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } health_report { overall_health: UNKNOWN } } }
I20260924 19:15:57.384006 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.376953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.11.18.67:39679)
0924 19:15:57.377150 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.377160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.377266 (+ 106us) server_negotiation.cc:407] Connection header received
0924 19:15:57.377594 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.377598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.377687 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.377827 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:15:57.378281 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.379116 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.381274 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.381809 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.381926 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.382587 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:15:57.383203 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:15:57.383415 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.383590 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.383668 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 19:15:57.384022 25713 tablet.cc:2369] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:15:57.394207 25399 raft_consensus.cc:1261] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 1 FOLLOWER]: Refusing update from remote peer d990c3e9c0ce4aec94ec2fac52a08112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:15:57.394207 25533 raft_consensus.cc:1261] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 FOLLOWER]: Refusing update from remote peer d990c3e9c0ce4aec94ec2fac52a08112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:15:57.395313 25741 consensus_queue.cc:1035] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:15:57.395730 25738 consensus_queue.cc:1035] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:15:57.429014 25255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:15:57.429464 25255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:15:57.433300 25255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Sending DeleteTablet for 3 replicas of tablet ebf2840adb7b45fb9948440b7b01b2da
I20260924 19:15:57.434685 25647 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf2840adb7b45fb9948440b7b01b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:15:57 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:40826
I20260924 19:15:57.434723 25513 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf2840adb7b45fb9948440b7b01b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:15:57 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:47382
I20260924 19:15:57.435478 25379 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf2840adb7b45fb9948440b7b01b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:15:57 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:34088
I20260924 19:15:57.435708 25753 tablet_replica.cc:330] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: stopping tablet replica
I20260924 19:15:57.436314 25753 raft_consensus.cc:2229] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:15:57.437222 25753 raft_consensus.cc:2258] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:15:57.437757 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 25316
I20260924 19:15:57.447384 25755 tablet_replica.cc:330] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: stopping tablet replica
I20260924 19:15:57.447422 25753 ts_tablet_manager.cc:1905] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:15:57.448113 25755 raft_consensus.cc:2229] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:15:57.448732 25755 raft_consensus.cc:2258] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:15:57.451081 25755 ts_tablet_manager.cc:1905] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:15:57.455232 25754 tablet_replica.cc:330] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: stopping tablet replica
I20260924 19:15:57.455466 25753 ts_tablet_manager.cc:1918] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:15:57.455662 25753 log.cc:1196] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/ebf2840adb7b45fb9948440b7b01b2da
I20260924 19:15:57.455863 25754 raft_consensus.cc:2229] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:15:57.456296 25753 ts_tablet_manager.cc:1939] T ebf2840adb7b45fb9948440b7b01b2da P d990c3e9c0ce4aec94ec2fac52a08112: Deleting consensus metadata
I20260924 19:15:57.456488 25754 raft_consensus.cc:2258] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:15:57.458209 25240 catalog_manager.cc:4841] TS d990c3e9c0ce4aec94ec2fac52a08112 (127.11.18.67:39679): tablet ebf2840adb7b45fb9948440b7b01b2da (table test-table [id=70951eac45c040a68e8dd6e1f0caa959]) successfully deleted
I20260924 19:15:57.459445 25755 ts_tablet_manager.cc:1918] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:15:57.459677 25755 log.cc:1196] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/ebf2840adb7b45fb9948440b7b01b2da
I20260924 19:15:57.460211 25754 ts_tablet_manager.cc:1905] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:15:57.460407 25755 ts_tablet_manager.cc:1939] T ebf2840adb7b45fb9948440b7b01b2da P 04c80d426b8d469cbf80e59ae5a054d6: Deleting consensus metadata
I20260924 19:15:57.462260 25240 catalog_manager.cc:4841] TS 04c80d426b8d469cbf80e59ae5a054d6 (127.11.18.65:38773): tablet ebf2840adb7b45fb9948440b7b01b2da (table test-table [id=70951eac45c040a68e8dd6e1f0caa959]) successfully deleted
Sep 24 19:15:57 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:15:57.466718 25754 ts_tablet_manager.cc:1918] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:15:57.466883 25754 log.cc:1196] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/ebf2840adb7b45fb9948440b7b01b2da
I20260924 19:15:57.467483 25754 ts_tablet_manager.cc:1939] T ebf2840adb7b45fb9948440b7b01b2da P 1f9a3b8824854431a8859c3b9d9814e3: Deleting consensus metadata
I20260924 19:15:57.470165 25241 catalog_manager.cc:4841] TS 1f9a3b8824854431a8859c3b9d9814e3 (127.11.18.66:33769): tablet ebf2840adb7b45fb9948440b7b01b2da (table test-table [id=70951eac45c040a68e8dd6e1f0caa959]) successfully deleted
I20260924 19:15:57.474340 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.439222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.11.18.65:38773)
0924 19:15:57.439475 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.439485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.447105 (+ 7620us) server_negotiation.cc:407] Connection header received
0924 19:15:57.448249 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.448258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.448407 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.448497 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:57.453144 (+ 4647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.454540 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.459740 (+ 5200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.460512 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.460645 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.467100 (+ 6455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:57.467121 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:57.467142 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:57.467225 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:57.471038 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.472023 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.472031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.472034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.472181 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.472636 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.472639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.472641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.473043 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:57.473278 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.473760 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.473965 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:15:57.475098 25439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:57.575327 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 25316
I20260924 19:15:57.602733 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 25450
Sep 24 19:15:57 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:15:57.627691 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.603683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.11.18.66:33769)
0924 19:15:57.603942 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.603949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.603974 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:15:57.604045 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.604053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.604143 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.604273 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:57.605815 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.607090 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.608801 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.609708 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.609893 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.619726 (+ 9833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:57.619750 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:57.619775 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:57.619837 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:57.624188 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.625131 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.625142 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.625146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.625317 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.625776 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.625782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.625786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.626323 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:57.626648 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.627020 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.627278 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 19:15:57.628441 25573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:57.720584 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 25450
I20260924 19:15:57.744922 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 25584
Sep 24 19:15:57 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:15:57.765972 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:57.745942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.11.18.67:39679)
0924 19:15:57.746147 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:57.746152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:57.746420 (+ 268us) server_negotiation.cc:407] Connection header received
0924 19:15:57.746559 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:57.746564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:57.746663 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:57.746760 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:57.748496 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.749580 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.751603 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:57.752141 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:57.752284 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:57.758141 (+ 5857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:57.758154 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:57.758169 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:57.758219 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:57.762289 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.763319 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.763331 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.763336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.763536 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:57.764136 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:57.764140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:57.764143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:57.764571 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:57.764916 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:57.765249 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 19:15:57.765551 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:15:57.766786 25707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:57.894906 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 25584
I20260924 19:15:57.915167 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 25223
I20260924 19:15:57.916450 25285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:15:58.030956 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 25223
I20260924 19:15:58.053841 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42123
--webserver_interface=127.11.18.126
--webserver_port=34495
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:15:58.406277 25761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:58.406576 25761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:58.406759 25761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:58.414373 25761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:15:58.414499 25761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:58.414572 25761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:58.414662 25761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:15:58.414765 25761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:15:58.425395 25761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=34495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:58.426872 25761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:58.428283 25761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:58.452440 25768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:58.452786 25770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:58.452639 25767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:58.454397 25761 server_base.cc:1034] running on GCE node
I20260924 19:15:58.455243 25761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:58.456898 25761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:58.458137 25761 hybrid_clock.cc:648] HybridClock initialized: now 1790277358457999 us; error 124 us; skew 500 ppm
I20260924 19:15:58.458654 25761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:58 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277358, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:58.462647 25761 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:15:58.464645 25761 webserver.cc:460] Webserver started at http://127.11.18.126:34495/ using document root <none> and password file <none>
I20260924 19:15:58.465677 25761 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:58.465849 25761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:58.472388 25761 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 19:15:58.475934 25777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:58.477810 25761 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:15:58.478049 25761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "38e256758cfc4a9fb5b42d36b8b467dc"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "a01930c62c1da913ba8357c32d560d90"
server_key_iv: "a50df667e94d31f7b01b22045511e2eb"
server_key_version: "encryptionkey@0"
I20260924 19:15:58.479306 25761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:58.509418 25761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:58.514937 25761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:58.515211 25761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:58.536284 25761 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:42123
I20260924 19:15:58.536306 25828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:42123 every 8 connection(s)
I20260924 19:15:58.538098 25761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:15:58.539918 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 25761
I20260924 19:15:58.540814 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:38773
--local_ip_for_outbound_sockets=127.11.18.65
--tserver_master_addrs=127.11.18.126:42123
--webserver_port=42825
--webserver_interface=127.11.18.65
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:15:58.545907 25829 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:15:58.549515 25829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Bootstrap starting.
I20260924 19:15:58.572219 25829 log.cc:824] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:15:58.586195 25829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:15:58.586779 25829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Bootstrap complete.
I20260924 19:15:58.596190 25829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:15:58.597123 25829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e256758cfc4a9fb5b42d36b8b467dc, State: Initialized, Role: FOLLOWER
I20260924 19:15:58.597787 25829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:15:58.598052 25829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:15:58.598189 25829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:15:58.598373 25829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:15:58.602856 25829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:15:58.603479 25829 leader_election.cc:302] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38e256758cfc4a9fb5b42d36b8b467dc; no voters:
I20260924 19:15:58.604214 25829 leader_election.cc:288] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:15:58.604391 25833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:15:58.605851 25833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 2 LEADER]: Becoming Leader. State: Replica: 38e256758cfc4a9fb5b42d36b8b467dc, State: Running, Role: LEADER
I20260924 19:15:58.606527 25833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:15:58.606923 25829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:15:58.609354 25835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38e256758cfc4a9fb5b42d36b8b467dc. Latest consensus state: current_term: 2 leader_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } } }
I20260924 19:15:58.609781 25835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:58.609472 25834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } } }
I20260924 19:15:58.610069 25834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: This master's current role is: LEADER
I20260924 19:15:58.622205 25840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:15:58.632987 25840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:15:58.634773 25840 catalog_manager.cc:1259] Loaded cluster ID: 14f0383ddb43424dbb91f60e966aa0bd
I20260924 19:15:58.634982 25840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:15:58.639175 25840 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 19:15:58.657545 25840 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 19:15:58.677573 25840 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:15:58.679570 25840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Loaded TSK: 0
I20260924 19:15:58.681203 25840 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:15:58.936256 25831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:58.936635 25831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:58.936820 25831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:58.944339 25831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:58.944509 25831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:58.944712 25831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:15:58.954988 25831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:38773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=42825
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:58.956425 25831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:58.957784 25831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:58.972061 25856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:58.972268 25857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:58.973093 25831 server_base.cc:1034] running on GCE node
W20260924 19:15:58.973606 25859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:58.974318 25831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:58.975502 25831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:58.976739 25831 hybrid_clock.cc:648] HybridClock initialized: now 1790277358976636 us; error 97 us; skew 500 ppm
I20260924 19:15:58.977090 25831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:58 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277358, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:58.989650 25831 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:15:58.991822 25831 webserver.cc:460] Webserver started at http://127.11.18.65:42825/ using document root <none> and password file <none>
I20260924 19:15:58.992865 25831 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:58.993042 25831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:59.000721 25831 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:15:59.004670 25866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:59.006605 25831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 19:15:59.006824 25831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "04c80d426b8d469cbf80e59ae5a054d6"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "e92471f99c002ed2c61d798577ee36a2"
server_key_iv: "ee0d692104a6308be9fc1b9dd52b858e"
server_key_version: "encryptionkey@0"
I20260924 19:15:59.008364 25831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:59.043291 25831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:59.049449 25831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:59.049767 25831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:59.051050 25831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:59.052814 25831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:59.052940 25831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:59.053041 25831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:59.053117 25831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:59.091938 25831 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:38773
I20260924 19:15:59.092077 25978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:38773 every 8 connection(s)
I20260924 19:15:59.093888 25831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:15:59.096879 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 25831
I20260924 19:15:59.097752 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:33769
--local_ip_for_outbound_sockets=127.11.18.66
--tserver_master_addrs=127.11.18.126:42123
--webserver_port=42755
--webserver_interface=127.11.18.66
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:15:59 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277358, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:59.123165 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:59.097236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:42709)
0924 19:15:59.098270 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:59.098318 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:59.100135 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:59.102048 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:59.102060 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:59.102746 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:59.103335 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:59.103356 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:59.105820 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:59.105829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:59.108441 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:59.108463 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:59.109674 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:59.109686 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:59.110102 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:59.110927 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:59.110974 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:59.114120 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:59.118200 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:59.118210 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:59.118227 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:59.118724 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:59.119240 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:59.119245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:59.119249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:59.119470 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:59.120279 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:59.120288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:59.120661 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 19:15:59.120958 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 19:15:59.121700 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":486,"threads_started":1}
I20260924 19:15:59.123173 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:59.097125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:42709 (local address 127.11.18.126:42123)
0924 19:15:59.097868 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:59.097890 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:59.098488 (+ 598us) server_negotiation.cc:407] Connection header received
0924 19:15:59.100797 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:59.100822 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:59.101370 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:59.101839 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:59.103620 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:59.105609 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:59.108644 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:59.109399 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:59.109569 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:59.114459 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:59.114472 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:59.114493 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:59.114545 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:59.118012 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:59.118895 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:59.118904 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:59.118910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:59.119100 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:59.119598 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:59.119606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:59.119609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:59.120119 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:59.120335 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:59.121198 (+ 863us) server_negotiation.cc:299] Negotiation successful
0924 19:15:59.121791 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
I20260924 19:15:59.126708 25979 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42123
I20260924 19:15:59.127067 25979 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:59.127880 25979 heartbeater.cc:511] Master 127.11.18.126:42123 requested a full tablet report, sending...
I20260924 19:15:59.130530 25794 ts_manager.cc:194] Registered new tserver with Master: 04c80d426b8d469cbf80e59ae5a054d6 (127.11.18.65:38773)
I20260924 19:15:59.132143 25794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:42709
W20260924 19:15:59.454516 25983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:59.454859 25983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:59.455022 25983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:59.462445 25983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:59.462606 25983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:59.462796 25983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:15:59.472811 25983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:33769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=42755
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:59.474236 25983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:59.475561 25983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:15:59.490007 25991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:59.489971 25989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:15:59.490638 25988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:15:59.490952 25983 server_base.cc:1034] running on GCE node
I20260924 19:15:59.491744 25983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:15:59.492918 25983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:15:59.494195 25983 hybrid_clock.cc:648] HybridClock initialized: now 1790277359494150 us; error 64 us; skew 500 ppm
I20260924 19:15:59.494626 25983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:15:59 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277359, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:15:59.498320 25983 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:15:59.500078 25983 webserver.cc:460] Webserver started at http://127.11.18.66:42755/ using document root <none> and password file <none>
I20260924 19:15:59.500844 25983 fs_manager.cc:359] Metadata directory not provided
I20260924 19:15:59.501019 25983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:15:59.507087 25983 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:15:59.510105 25998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:59.511672 25983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:15:59.511914 25983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "1f9a3b8824854431a8859c3b9d9814e3"
format_stamp: "Formatted at 2026-09-24 19:15:56 on dist-test-slave-0lgr"
server_key: "efb71a580c94e2aa7f8f8b812cb4d23a"
server_key_iv: "f790385836c387b198b3d2a80b2b786b"
server_key_version: "encryptionkey@0"
I20260924 19:15:59.513008 25983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:15:59.536684 25983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:15:59.541778 25983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:15:59.542112 25983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:15:59.543227 25983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:15:59.544839 25983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:15:59.544988 25983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:59.545147 25983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:15:59.545245 25983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:15:59.584939 25983 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:33769
I20260924 19:15:59.585032 26110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:33769 every 8 connection(s)
I20260924 19:15:59.586493 25983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:15:59.596875 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 25983
I20260924 19:15:59.597640 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:39679
--local_ip_for_outbound_sockets=127.11.18.67
--tserver_master_addrs=127.11.18.126:42123
--webserver_port=38787
--webserver_interface=127.11.18.67
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:15:59 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277359, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:15:59.610409 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:59.589208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:55617 (local address 127.11.18.126:42123)
0924 19:15:59.589458 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:15:59.589464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:15:59.590591 (+ 1127us) server_negotiation.cc:407] Connection header received
0924 19:15:59.592676 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:15:59.592681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:15:59.592787 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:15:59.592883 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:15:59.594656 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:59.595378 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:59.597413 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:15:59.597958 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:15:59.598057 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:59.602807 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:15:59.602812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:15:59.602816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:15:59.602855 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:15:59.606253 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:59.607529 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:59.607534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:59.607537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:59.607685 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:15:59.608293 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:15:59.608297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:15:59.608299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:15:59.608700 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:15:59.608857 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:15:59.609953 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0924 19:15:59.610154 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:15:59.612414 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:15:59.589646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:55617)
0924 19:15:59.590416 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 19:15:59.590458 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:15:59.592357 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:15:59.593163 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:15:59.593172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:15:59.593892 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:15:59.594472 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:59.594490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:59.595560 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:59.595564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:59.597206 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:15:59.597224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:15:59.598200 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:15:59.598211 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:15:59.598563 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:15:59.599431 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:15:59.599470 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:15:59.602487 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:15:59.606461 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:59.606475 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:59.606501 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:59.607322 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:59.607860 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:15:59.607869 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:15:59.607876 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:15:59.608149 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:15:59.608852 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:15:59.608864 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:15:59.609473 (+ 609us) client_negotiation.cc:769] Sending connection context
0924 19:15:59.609825 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 19:15:59.610510 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":291,"threads_started":1}
I20260924 19:15:59.614254 26111 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42123
I20260924 19:15:59.614619 26111 heartbeater.cc:464] Registering TS with master...
I20260924 19:15:59.615552 26111 heartbeater.cc:511] Master 127.11.18.126:42123 requested a full tablet report, sending...
I20260924 19:15:59.617264 25794 ts_manager.cc:194] Registered new tserver with Master: 1f9a3b8824854431a8859c3b9d9814e3 (127.11.18.66:33769)
I20260924 19:15:59.618326 25794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:55617
W20260924 19:15:59.962374 26114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:15:59.962746 26114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:15:59.962940 26114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:15:59.971593 26114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:15:59.971768 26114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:15:59.971959 26114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:15:59.982831 26114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:39679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=38787
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:15:59.984267 26114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:15:59.985723 26114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:00.000854 26120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:00.000711 26119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:00.000887 26122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:00.002140 26114 server_base.cc:1034] running on GCE node
I20260924 19:16:00.003122 26114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:00.004367 26114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:00.005640 26114 hybrid_clock.cc:648] HybridClock initialized: now 1790277360005600 us; error 31 us; skew 500 ppm
I20260924 19:16:00.006062 26114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:00 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:00.010397 26114 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:16:00.012056 26114 webserver.cc:460] Webserver started at http://127.11.18.67:38787/ using document root <none> and password file <none>
I20260924 19:16:00.012769 26114 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:00.012892 26114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:00.019409 26114 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:16:00.023217 26129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:00.024821 26114 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:16:00.025007 26114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "d990c3e9c0ce4aec94ec2fac52a08112"
format_stamp: "Formatted at 2026-09-24 19:15:57 on dist-test-slave-0lgr"
server_key: "8a44059a36e2719a9886165890ebff41"
server_key_iv: "4f69affc7f6e909512171e5cbdec9758"
server_key_version: "encryptionkey@0"
I20260924 19:16:00.026109 26114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:00.041858 26114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:00.046425 26114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:00.046717 26114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:00.047737 26114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:00.049230 26114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:00.049367 26114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:00.049482 26114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:00.049590 26114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:00.086499 26114 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:39679
I20260924 19:16:00.086627 26241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:39679 every 8 connection(s)
I20260924 19:16:00.088281 26114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:16:00.097946 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 26114
Sep 24 19:16:00 dist-test-slave-0lgr krb5kdc[25210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277360, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:16:00.115636 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.091260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:58403 (local address 127.11.18.126:42123)
0924 19:16:00.091498 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.091505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.092891 (+ 1386us) server_negotiation.cc:407] Connection header received
0924 19:16:00.095658 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.095663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.095740 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.095845 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:00.098123 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.099147 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.101727 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.102194 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.102353 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.107235 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:00.107240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:00.107244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:00.107302 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:00.111385 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.112570 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.112574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.112577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.112724 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.113522 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.113531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.113535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.113932 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:00.114097 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.115168 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.115385 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:16:00.116855 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.091777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:58403)
0924 19:16:00.092675 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:00.092731 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:00.095287 (+ 2556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:00.096136 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:00.096148 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:00.096959 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:16:00.097867 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.097894 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.099325 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.099330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.101548 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.101564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.102450 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.102461 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.102746 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.103628 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:16:00.103663 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:16:00.106958 (+ 3295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:16:00.111571 (+ 4613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:00.111582 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:00.111603 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:00.112342 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:00.112911 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:00.112916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:00.112920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:00.113186 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:00.114068 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:16:00.114076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:16:00.114597 (+ 521us) client_negotiation.cc:769] Sending connection context
0924 19:16:00.114972 (+ 375us) client_negotiation.cc:240] Negotiation successful
0924 19:16:00.115589 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":339,"threads_started":1}
I20260924 19:16:00.118495 26242 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42123
I20260924 19:16:00.118863 26242 heartbeater.cc:464] Registering TS with master...
I20260924 19:16:00.119087 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.099111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.11.18.126:42123)
0924 19:16:00.100421 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.100428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.100451 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:16:00.100508 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.100514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.100597 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.100729 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:00.103028 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.104292 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.106471 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.107234 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.107419 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.111572 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:00.111579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:00.111584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:00.111626 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:00.115691 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.116875 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.116883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.116887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.117028 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.117468 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.117472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.117475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.117882 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:00.118170 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.118445 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.118719 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":928,"threads_started":1}
I20260924 19:16:00.119809 26242 heartbeater.cc:511] Master 127.11.18.126:42123 requested a full tablet report, sending...
I20260924 19:16:00.121539 25793 ts_manager.cc:194] Registered new tserver with Master: d990c3e9c0ce4aec94ec2fac52a08112 (127.11.18.67:39679)
I20260924 19:16:00.122656 25793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:58403
I20260924 19:16:00.134974 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:16:00.135836 25979 heartbeater.cc:503] Master 127.11.18.126:42123 was elected leader, sending a full tablet report...
I20260924 19:16:00.142427 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.136009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.11.18.126:42123)
0924 19:16:00.136240 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.136247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.136827 (+ 580us) server_negotiation.cc:407] Connection header received
0924 19:16:00.137087 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.137090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.137228 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.137432 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:16:00.138014 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.138724 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.140245 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.140680 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.140776 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.141251 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:16:00.141872 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:16:00.142019 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.142158 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.142226 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 19:16:00.143322 25793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:16:00.150818 25793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:16:00.180599 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.163784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:38773 (local address 127.0.0.1:34106)
0924 19:16:00.164780 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:00.164844 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:00.165048 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:00.170375 (+ 5327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:00.170387 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:00.170432 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:00.171072 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.171091 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.174169 (+ 3078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.174179 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.177464 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.177481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.179619 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.179625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.179850 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.179908 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 19:16:00.180016 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:16:00.180226 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":544,"threads_started":1}
I20260924 19:16:00.181766 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.164180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.11.18.65:38773)
0924 19:16:00.169393 (+ 5213us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.169406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.169447 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:16:00.169592 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.169605 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.170032 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.171375 (+ 1343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:00.171703 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.173950 (+ 2247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.177655 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.179384 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.180875 (+ 1491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.181144 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.181224 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.181437 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4903,"thread_start_us":4665,"threads_started":1}
I20260924 19:16:00.186077 26257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.164236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:33769 (local address 127.0.0.1:47402)
0924 19:16:00.175635 (+ 11399us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:00.175665 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:00.175837 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:00.178092 (+ 2255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:00.178099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:00.178126 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:00.178566 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.178577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.181528 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.181533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.183155 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.183169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.185117 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.185124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.185291 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.185366 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 19:16:00.185489 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 19:16:00.185647 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11201,"mutex_wait_us":2,"thread_start_us":12366,"threads_started":1}
I20260924 19:16:00.186277 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.164798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.11.18.66:33769)
0924 19:16:00.177641 (+ 12843us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.177648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.177668 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:16:00.177746 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.177751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.177909 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.178108 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:00.178736 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.181293 (+ 2557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.183369 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.184896 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.185060 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.185268 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.185817 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.185996 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12665,"thread_start_us":12558,"threads_started":1}
I20260924 19:16:00.185808 25914 tablet_service.cc:1467] Processing CreateTablet for tablet d0cda2d506ca4ee59ec558756a7a7b0a (DEFAULT_TABLE table=test-table [id=4ca6bd4593f647a38f049992e2666d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:00.186489 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.165555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.11.18.67:39679)
0924 19:16:00.166481 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.166494 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.177049 (+ 10555us) server_negotiation.cc:407] Connection header received
0924 19:16:00.177426 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.177435 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.177707 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.177906 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:00.179095 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.180634 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.183992 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.185353 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.185482 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.185688 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.185959 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.186151 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":602,"threads_started":1}
I20260924 19:16:00.187100 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.166028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:39679 (local address 127.0.0.1:40854)
0924 19:16:00.176915 (+ 10887us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:00.176934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:00.177188 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:00.178224 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:00.178232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:00.178260 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:00.178832 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.178843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.180821 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.180826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.183790 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.183806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.185571 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.185577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.185730 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.185774 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:16:00.186680 (+ 906us) client_negotiation.cc:240] Negotiation successful
0924 19:16:00.186815 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10717,"mutex_wait_us":63,"thread_start_us":9929,"threads_started":1}
I20260924 19:16:00.187515 25914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cda2d506ca4ee59ec558756a7a7b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:00.190382 26177 tablet_service.cc:1467] Processing CreateTablet for tablet d0cda2d506ca4ee59ec558756a7a7b0a (DEFAULT_TABLE table=test-table [id=4ca6bd4593f647a38f049992e2666d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:00.190279 26046 tablet_service.cc:1467] Processing CreateTablet for tablet d0cda2d506ca4ee59ec558756a7a7b0a (DEFAULT_TABLE table=test-table [id=4ca6bd4593f647a38f049992e2666d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:00.191541 26177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cda2d506ca4ee59ec558756a7a7b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:00.191672 26046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cda2d506ca4ee59ec558756a7a7b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:00.203763 26258 tablet_bootstrap.cc:492] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Bootstrap starting.
I20260924 19:16:00.205979 26259 tablet_bootstrap.cc:492] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Bootstrap starting.
I20260924 19:16:00.208541 26260 tablet_bootstrap.cc:492] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Bootstrap starting.
I20260924 19:16:00.208925 26258 tablet_bootstrap.cc:654] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:00.209754 26259 tablet_bootstrap.cc:654] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:00.210357 26258 log.cc:824] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:00.211117 26259 log.cc:824] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:00.212394 26260 tablet_bootstrap.cc:654] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:00.213438 26260 log.cc:824] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:00.214284 26258 tablet_bootstrap.cc:492] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: No bootstrap required, opened a new log
I20260924 19:16:00.214567 26258 ts_tablet_manager.cc:1397] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 19:16:00.216434 26259 tablet_bootstrap.cc:492] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: No bootstrap required, opened a new log
I20260924 19:16:00.216760 26259 ts_tablet_manager.cc:1397] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260924 19:16:00.223273 26258 raft_consensus.cc:348] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.223664 26258 raft_consensus.cc:374] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:00.223757 26258 raft_consensus.cc:729] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d990c3e9c0ce4aec94ec2fac52a08112, State: Initialized, Role: FOLLOWER
I20260924 19:16:00.224359 26260 tablet_bootstrap.cc:492] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: No bootstrap required, opened a new log
I20260924 19:16:00.224439 26258 consensus_queue.cc:260] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.224644 26260 ts_tablet_manager.cc:1397] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260924 19:16:00.225597 26242 heartbeater.cc:503] Master 127.11.18.126:42123 was elected leader, sending a full tablet report...
I20260924 19:16:00.226257 26258 ts_tablet_manager.cc:1428] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260924 19:16:00.229092 26259 raft_consensus.cc:348] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.229571 26259 raft_consensus.cc:374] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:00.229667 26259 raft_consensus.cc:729] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c80d426b8d469cbf80e59ae5a054d6, State: Initialized, Role: FOLLOWER
I20260924 19:16:00.230366 26259 consensus_queue.cc:260] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.232332 26259 ts_tablet_manager.cc:1428] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 19:16:00.235029 26260 raft_consensus.cc:348] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.235422 26260 raft_consensus.cc:374] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:00.235526 26260 raft_consensus.cc:729] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9a3b8824854431a8859c3b9d9814e3, State: Initialized, Role: FOLLOWER
I20260924 19:16:00.236115 26260 consensus_queue.cc:260] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.237313 26111 heartbeater.cc:503] Master 127.11.18.126:42123 was elected leader, sending a full tablet report...
I20260924 19:16:00.238004 26260 ts_tablet_manager.cc:1428] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
W20260924 19:16:00.338665 26112 tablet.cc:2369] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:16:00.340132 26243 tablet.cc:2369] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:16:00.346505 25980 tablet.cc:2369] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:16:00.389930 26266 raft_consensus.cc:482] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:16:00.390370 26266 raft_consensus.cc:504] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.392288 26266 leader_election.cc:288] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c80d426b8d469cbf80e59ae5a054d6 (127.11.18.65:38773), d990c3e9c0ce4aec94ec2fac52a08112 (127.11.18.67:39679)
I20260924 19:16:00.400336 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.392952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:48093 (local address 127.11.18.65:38773)
0924 19:16:00.393168 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.393173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.393908 (+ 735us) server_negotiation.cc:407] Connection header received
0924 19:16:00.394290 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.394300 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.394412 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.394486 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:00.395287 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.396389 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.398399 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.399276 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.399489 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.399578 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.399889 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.400044 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:16:00.400549 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.392866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:38773 (local address 127.11.18.66:48093)
0924 19:16:00.393758 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:00.393781 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:00.394050 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:00.394714 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:00.394718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:00.394741 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:00.395135 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.395144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.396560 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.396564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.398206 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.398220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.399525 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.399532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.399711 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.399737 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 19:16:00.399859 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 19:16:00.400033 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":25,"spinlock_wait_cycles":45312,"thread_start_us":488,"threads_started":1}
I20260924 19:16:00.401219 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.394234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:54257 (local address 127.11.18.67:39679)
0924 19:16:00.394489 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.394497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.394852 (+ 355us) server_negotiation.cc:407] Connection header received
0924 19:16:00.395123 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.395132 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.395216 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.395318 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:00.395967 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.396935 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.398868 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.399905 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.400142 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.400221 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.400766 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.400934 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 19:16:00.401297 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.393885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:39679 (local address 127.11.18.66:54257)
0924 19:16:00.394726 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:00.394742 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:00.394964 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:00.395445 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:00.395449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:00.395471 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:00.395791 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.395799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.397091 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.397097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.398694 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:00.398707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.400161 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.400186 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:00.400431 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.400501 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 19:16:00.400756 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 19:16:00.400945 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":1,"thread_start_us":511,"threads_started":1}
I20260924 19:16:00.401916 25934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cda2d506ca4ee59ec558756a7a7b0a" candidate_uuid: "1f9a3b8824854431a8859c3b9d9814e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c80d426b8d469cbf80e59ae5a054d6" is_pre_election: true
I20260924 19:16:00.402621 25934 raft_consensus.cc:2454] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9a3b8824854431a8859c3b9d9814e3 in term 0.
I20260924 19:16:00.402601 26197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cda2d506ca4ee59ec558756a7a7b0a" candidate_uuid: "1f9a3b8824854431a8859c3b9d9814e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" is_pre_election: true
I20260924 19:16:00.403209 26197 raft_consensus.cc:2454] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9a3b8824854431a8859c3b9d9814e3 in term 0.
I20260924 19:16:00.403698 26000 leader_election.cc:302] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c80d426b8d469cbf80e59ae5a054d6, 1f9a3b8824854431a8859c3b9d9814e3; no voters:
I20260924 19:16:00.404187 26266 raft_consensus.cc:2781] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:16:00.404361 26266 raft_consensus.cc:482] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:16:00.404475 26266 raft_consensus.cc:3037] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:00.406765 26266 raft_consensus.cc:504] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.407431 26266 leader_election.cc:288] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [CANDIDATE]: Term 1 election: Requested vote from peers 04c80d426b8d469cbf80e59ae5a054d6 (127.11.18.65:38773), d990c3e9c0ce4aec94ec2fac52a08112 (127.11.18.67:39679)
I20260924 19:16:00.407959 26197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cda2d506ca4ee59ec558756a7a7b0a" candidate_uuid: "1f9a3b8824854431a8859c3b9d9814e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d990c3e9c0ce4aec94ec2fac52a08112"
I20260924 19:16:00.407989 25934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cda2d506ca4ee59ec558756a7a7b0a" candidate_uuid: "1f9a3b8824854431a8859c3b9d9814e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c80d426b8d469cbf80e59ae5a054d6"
I20260924 19:16:00.408177 26197 raft_consensus.cc:3037] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:00.408280 25934 raft_consensus.cc:3037] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:00.410564 26197 raft_consensus.cc:2454] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9a3b8824854431a8859c3b9d9814e3 in term 1.
I20260924 19:16:00.410913 25934 raft_consensus.cc:2454] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9a3b8824854431a8859c3b9d9814e3 in term 1.
I20260924 19:16:00.410974 26000 leader_election.cc:302] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9a3b8824854431a8859c3b9d9814e3, d990c3e9c0ce4aec94ec2fac52a08112; no voters:
I20260924 19:16:00.411309 26266 raft_consensus.cc:2781] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:16:00.411805 26266 raft_consensus.cc:686] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9a3b8824854431a8859c3b9d9814e3, State: Running, Role: LEADER
I20260924 19:16:00.412289 26266 consensus_queue.cc:237] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } }
I20260924 19:16:00.417805 25793 catalog_manager.cc:5495] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9a3b8824854431a8859c3b9d9814e3 (127.11.18.66). New cstate: current_term: 1 leader_uuid: "1f9a3b8824854431a8859c3b9d9814e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9a3b8824854431a8859c3b9d9814e3" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 33769 } health_report { overall_health: HEALTHY } } }
I20260924 19:16:00.488904 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.482532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.11.18.66:33769)
0924 19:16:00.482764 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.482773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.482798 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:16:00.482914 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.482918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.483078 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.483248 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:16:00.483713 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.484605 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.486436 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.486903 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.487055 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.487622 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:16:00.488181 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:16:00.488353 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.488449 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.488544 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:16:00.498231 26197 raft_consensus.cc:1261] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 FOLLOWER]: Refusing update from remote peer 1f9a3b8824854431a8859c3b9d9814e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:16:00.498504 25934 raft_consensus.cc:1261] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 1 FOLLOWER]: Refusing update from remote peer 1f9a3b8824854431a8859c3b9d9814e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:16:00.499282 26269 consensus_queue.cc:1035] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d990c3e9c0ce4aec94ec2fac52a08112" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:16:00.499698 26266 consensus_queue.cc:1035] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c80d426b8d469cbf80e59ae5a054d6" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:16:00.513502 26273 mvcc.cc:205] Tried to move back new op lower bound from 7332976068590174208 to 7332976068253204480. Current Snapshot: MvccSnapshot[applied={T|T < 7332976068590174208}]
I20260924 19:16:00.528113 25794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:16:00.528437 25794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:16:00.532752 25794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Sending DeleteTablet for 3 replicas of tablet d0cda2d506ca4ee59ec558756a7a7b0a
I20260924 19:16:00.534197 26046 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cda2d506ca4ee59ec558756a7a7b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:00 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:47402
I20260924 19:16:00.534327 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 25831
I20260924 19:16:00.534526 26177 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cda2d506ca4ee59ec558756a7a7b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:00 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:40854
I20260924 19:16:00.535156 25914 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cda2d506ca4ee59ec558756a7a7b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:00 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:34106
I20260924 19:16:00.536155 26284 tablet_replica.cc:330] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: stopping tablet replica
I20260924 19:16:00.536803 26284 raft_consensus.cc:2229] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:16:00.537566 26284 raft_consensus.cc:2258] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:16:00.541781 26284 ts_tablet_manager.cc:1905] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:16:00.550478 26283 tablet_replica.cc:330] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: stopping tablet replica
I20260924 19:16:00.551129 26283 raft_consensus.cc:2229] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:16:00.551807 26283 raft_consensus.cc:2258] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:16:00.558112 26284 ts_tablet_manager.cc:1918] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:16:00.559544 26284 log.cc:1196] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/d0cda2d506ca4ee59ec558756a7a7b0a
I20260924 19:16:00.563771 26284 ts_tablet_manager.cc:1939] T d0cda2d506ca4ee59ec558756a7a7b0a P 04c80d426b8d469cbf80e59ae5a054d6: Deleting consensus metadata
I20260924 19:16:00.563966 26283 ts_tablet_manager.cc:1905] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:16:00.565837 26282 tablet_replica.cc:330] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: stopping tablet replica
I20260924 19:16:00.566326 26282 raft_consensus.cc:2229] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:16:00.566752 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.535306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.11.18.65:38773)
0924 19:16:00.535502 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.535512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.536131 (+ 619us) server_negotiation.cc:407] Connection header received
0924 19:16:00.536204 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.536211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.536355 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.536456 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:00.538435 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.540049 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.542343 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.543069 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.543238 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.558468 (+ 15230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:00.558486 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:00.558508 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:00.558564 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:00.562869 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.564222 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.564232 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.564236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.564419 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.564842 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.564848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.564852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.565345 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:00.565675 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.566052 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.566283 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:16:00.567065 26282 raft_consensus.cc:2258] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:16:00.567773 25974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:00.570314 26282 ts_tablet_manager.cc:1905] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:16:00.571620 26283 ts_tablet_manager.cc:1918] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:16:00.571779 26283 log.cc:1196] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/d0cda2d506ca4ee59ec558756a7a7b0a
I20260924 19:16:00.572268 26283 ts_tablet_manager.cc:1939] T d0cda2d506ca4ee59ec558756a7a7b0a P d990c3e9c0ce4aec94ec2fac52a08112: Deleting consensus metadata
I20260924 19:16:00.574123 25779 catalog_manager.cc:4841] TS d990c3e9c0ce4aec94ec2fac52a08112 (127.11.18.67:39679): tablet d0cda2d506ca4ee59ec558756a7a7b0a (table test-table [id=4ca6bd4593f647a38f049992e2666d5a]) successfully deleted
I20260924 19:16:00.581074 26282 ts_tablet_manager.cc:1918] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:16:00.581275 26282 log.cc:1196] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/d0cda2d506ca4ee59ec558756a7a7b0a
I20260924 19:16:00.581908 26282 ts_tablet_manager.cc:1939] T d0cda2d506ca4ee59ec558756a7a7b0a P 1f9a3b8824854431a8859c3b9d9814e3: Deleting consensus metadata
I20260924 19:16:00.583829 25780 catalog_manager.cc:4841] TS 1f9a3b8824854431a8859c3b9d9814e3 (127.11.18.66:33769): tablet d0cda2d506ca4ee59ec558756a7a7b0a (table test-table [id=4ca6bd4593f647a38f049992e2666d5a]) successfully deleted
I20260924 19:16:00.693274 25779 catalog_manager.cc:4841] TS 04c80d426b8d469cbf80e59ae5a054d6 (127.11.18.65:38773): tablet d0cda2d506ca4ee59ec558756a7a7b0a (table test-table [id=4ca6bd4593f647a38f049992e2666d5a]) successfully deleted
I20260924 19:16:00.693390 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 25831
I20260924 19:16:00.720642 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 25983
I20260924 19:16:00.738785 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.721620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.11.18.66:33769)
0924 19:16:00.721802 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.721807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.721830 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:16:00.721891 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.721895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.721998 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.722092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:00.724003 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.725030 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.727065 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.727577 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.727724 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.731481 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:00.731502 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:00.731520 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:00.731567 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:00.735165 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.736296 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.736304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.736306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.736463 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.736914 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.736918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.736920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.737294 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:00.737617 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.737992 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.738168 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:16:00.739485 26106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:00.860754 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 25983
I20260924 19:16:00.882056 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 26114
I20260924 19:16:00.897629 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:00.882901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.11.18.67:39679)
0924 19:16:00.883097 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:00.883102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:00.883190 (+ 88us) server_negotiation.cc:407] Connection header received
0924 19:16:00.883404 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:00.883410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:00.883485 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:00.883600 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:00.885305 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.886201 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.887950 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:00.888370 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:00.888480 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:00.891565 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:00.891580 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:00.891596 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:00.891643 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:00.894578 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.895398 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.895406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.895408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.895581 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:00.895966 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:00.895970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:00.895972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:00.896400 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:00.896641 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:00.896976 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 19:16:00.897156 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:16:00.898357 26237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:00.991642 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 26114
I20260924 19:16:01.009223 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 25761
I20260924 19:16:01.010617 25824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:01.128055 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 25761
I20260924 19:16:01.152432 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42123
--webserver_interface=127.11.18.126
--webserver_port=34495
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:16:01.503008 26289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:01.503456 26289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:01.503641 26289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:01.512300 26289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:16:01.512615 26289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:01.512789 26289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:01.512912 26289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:16:01.513011 26289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:16:01.524353 26289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=34495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:01.525840 26289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:01.527240 26289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:01.540167 26294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:01.540134 26297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:01.540211 26295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:01.540807 26289 server_base.cc:1034] running on GCE node
I20260924 19:16:01.541754 26289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:01.543231 26289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:01.544502 26289 hybrid_clock.cc:648] HybridClock initialized: now 1790277361544436 us; error 62 us; skew 500 ppm
I20260924 19:16:01.544872 26289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:01 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277361, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:01.548599 26289 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:16:01.550273 26289 webserver.cc:460] Webserver started at http://127.11.18.126:34495/ using document root <none> and password file <none>
I20260924 19:16:01.551005 26289 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:01.551118 26289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:01.557574 26289 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 19:16:01.560779 26304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:01.562526 26289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:16:01.562739 26289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "38e256758cfc4a9fb5b42d36b8b467dc"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "a01930c62c1da913ba8357c32d560d90"
server_key_iv: "a50df667e94d31f7b01b22045511e2eb"
server_key_version: "encryptionkey@0"
I20260924 19:16:01.563897 26289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:01.582295 26289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:01.587358 26289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:01.587697 26289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:01.609638 26289 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:42123
I20260924 19:16:01.609681 26355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:42123 every 8 connection(s)
I20260924 19:16:01.611514 26289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:16:01.616912 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 26289
I20260924 19:16:01.618047 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:38773
--local_ip_for_outbound_sockets=127.11.18.65
--tserver_master_addrs=127.11.18.126:42123
--webserver_port=42825
--webserver_interface=127.11.18.65
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:16:01.619817 26356 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:16:01.623301 26356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Bootstrap starting.
I20260924 19:16:01.654642 26356 log.cc:824] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:01.679975 26356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:16:01.680806 26356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc: Bootstrap complete.
I20260924 19:16:01.697950 26356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:16:01.699373 26356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e256758cfc4a9fb5b42d36b8b467dc, State: Initialized, Role: FOLLOWER
I20260924 19:16:01.700184 26356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:16:01.700486 26356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:16:01.700626 26356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:16:01.700822 26356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:16:01.705679 26356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:16:01.706295 26356 leader_election.cc:302] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38e256758cfc4a9fb5b42d36b8b467dc; no voters:
I20260924 19:16:01.707098 26356 leader_election.cc:288] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:16:01.707261 26360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:16:01.708561 26360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [term 3 LEADER]: Becoming Leader. State: Replica: 38e256758cfc4a9fb5b42d36b8b467dc, State: Running, Role: LEADER
I20260924 19:16:01.709259 26360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } }
I20260924 19:16:01.709796 26356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:16:01.712344 26362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38e256758cfc4a9fb5b42d36b8b467dc. Latest consensus state: current_term: 3 leader_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } } }
I20260924 19:16:01.712769 26362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: This master's current role is: LEADER
I20260924 19:16:01.713428 26361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e256758cfc4a9fb5b42d36b8b467dc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42123 } } }
I20260924 19:16:01.713714 26361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e256758cfc4a9fb5b42d36b8b467dc [sys.catalog]: This master's current role is: LEADER
I20260924 19:16:01.724489 26367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:16:01.734138 26367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:16:01.736086 26367 catalog_manager.cc:1259] Loaded cluster ID: 14f0383ddb43424dbb91f60e966aa0bd
I20260924 19:16:01.736246 26367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 19:16:01.738410 26367 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f84844249ad google::LogMessage::Fail() at ??:0
@ 0x7f848442788a google::LogMessage::SendToLog() at ??:0
@ 0x7f84844244de google::LogMessage::Flush() at ??:0
@ 0x7f8484425a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260924 19:16:01.748020 26377 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f8499e57512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8499e54976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8499edc610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8499edc439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8487070345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f848707391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8487073799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f848704eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f848a3286db start_thread at ??:0
@ 0x7f84832be61f clone at ??:0
W20260924 19:16:02.087843 26358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:02.088178 26358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:02.088366 26358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:02.097075 26358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:02.097257 26358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:02.097517 26358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:16:02.110957 26358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:38773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=42825
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:02.112767 26358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:02.114696 26358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:02.131835 26398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:02.131870 26400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:02.132205 26397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:02.134131 26358 server_base.cc:1034] running on GCE node
I20260924 19:16:02.134994 26358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:02.136178 26358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:02.137532 26358 hybrid_clock.cc:648] HybridClock initialized: now 1790277362137455 us; error 79 us; skew 500 ppm
I20260924 19:16:02.137861 26358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:02 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277362, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:02.141701 26358 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:16:02.143559 26358 webserver.cc:460] Webserver started at http://127.11.18.65:42825/ using document root <none> and password file <none>
I20260924 19:16:02.144414 26358 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:02.144599 26358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:02.152854 26358 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:16:02.157948 26407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:02.160152 26358 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260924 19:16:02.160403 26358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "04c80d426b8d469cbf80e59ae5a054d6"
format_stamp: "Formatted at 2026-09-24 19:15:55 on dist-test-slave-0lgr"
server_key: "e92471f99c002ed2c61d798577ee36a2"
server_key_iv: "ee0d692104a6308be9fc1b9dd52b858e"
server_key_version: "encryptionkey@0"
I20260924 19:16:02.161619 26358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:02.185772 26358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:02.191797 26358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:02.192158 26358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:02.193516 26358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:02.195429 26358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:02.195571 26358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:02.195746 26358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:02.195853 26358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:02.239959 26358 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:38773
I20260924 19:16:02.240070 26519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:38773 every 8 connection(s)
I20260924 19:16:02.241678 26358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:16:02.249650 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:02.246430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:50887)
0924 19:16:02.247543 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:02.247914 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":458,"threads_started":1}
I20260924 19:16:02.250770 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 26358
I20260924 19:16:02.251623 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:33769
--local_ip_for_outbound_sockets=127.11.18.66
--tserver_master_addrs=127.11.18.126:42123
--webserver_port=42755
--webserver_interface=127.11.18.66
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:16:02.258337 26520 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:42123 (0 consecutive failures): Network error: Failed to ping master at 127.11.18.126:42123: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
I20260924 19:16:02.259761 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:02.259330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:49983)
0924 19:16:02.259516 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:02.259611 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 19:16:02.261256 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:02.260769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:54259)
0924 19:16:02.260968 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:02.261078 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:16:02.261668 26520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:16:02.629079 26523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:02.629455 26523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:02.629571 26523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:02.637984 26523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:02.638118 26523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:02.638245 26523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:16:02.649636 26523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:33769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=42755
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:02.651053 26523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:02.652467 26523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:02.667491 26528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:02.667590 26529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:02.668026 26531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:02.669102 26523 server_base.cc:1034] running on GCE node
I20260924 19:16:02.670027 26523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:02.671308 26523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:02.672569 26523 hybrid_clock.cc:648] HybridClock initialized: now 1790277362672529 us; error 35 us; skew 500 ppm
I20260924 19:16:02.672942 26523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:02 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277362, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:02.676569 26523 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:16:02.678309 26523 webserver.cc:460] Webserver started at http://127.11.18.66:42755/ using document root <none> and password file <none>
I20260924 19:16:02.679051 26523 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:02.679186 26523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:02.684968 26523 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:16:02.688292 26538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:02.690047 26523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:16:02.690243 26523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "1f9a3b8824854431a8859c3b9d9814e3"
format_stamp: "Formatted at 2026-09-24 19:15:56 on dist-test-slave-0lgr"
server_key: "efb71a580c94e2aa7f8f8b812cb4d23a"
server_key_iv: "f790385836c387b198b3d2a80b2b786b"
server_key_version: "encryptionkey@0"
I20260924 19:16:02.691375 26523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:02.712989 26523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:02.718537 26523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:02.718879 26523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:02.720151 26523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:02.721706 26523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:02.721841 26523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:02.721997 26523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:02.722080 26523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:02.760026 26523 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:33769
I20260924 19:16:02.760082 26650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:33769 every 8 connection(s)
I20260924 19:16:02.761804 26523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:16:02.767432 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 26523
I20260924 19:16:02.767552 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:02.765187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:54185)
0924 19:16:02.766083 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:02.766300 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":385,"threads_started":1}
I20260924 19:16:02.768237 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:39679
--local_ip_for_outbound_sockets=127.11.18.67
--tserver_master_addrs=127.11.18.126:42123
--webserver_port=38787
--webserver_interface=127.11.18.67
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:16:02.768333 26651 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:42123 (0 consecutive failures): Network error: Failed to ping master at 127.11.18.126:42123: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
I20260924 19:16:02.769526 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:02.769012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:37771)
0924 19:16:02.769228 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:02.769358 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 19:16:02.770684 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:02.770271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:34409)
0924 19:16:02.770446 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:02.770543 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 19:16:02.771052 26651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:16:03.138388 26654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:03.138746 26654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:03.138854 26654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:03.147437 26654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:03.147697 26654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:03.147919 26654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:16:03.159695 26654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:39679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=38787
--tserver_master_addrs=127.11.18.126:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:03.161087 26654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:03.162580 26654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:03.178158 26660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:03.178249 26659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:03.178210 26662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:03.179606 26654 server_base.cc:1034] running on GCE node
I20260924 19:16:03.180358 26654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:03.181610 26654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:03.182862 26654 hybrid_clock.cc:648] HybridClock initialized: now 1790277363182807 us; error 68 us; skew 500 ppm
I20260924 19:16:03.183252 26654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:03 dist-test-slave-0lgr krb5kdc[25210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277363, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:03.186980 26654 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:16:03.188879 26654 webserver.cc:460] Webserver started at http://127.11.18.67:38787/ using document root <none> and password file <none>
I20260924 19:16:03.189798 26654 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:03.189965 26654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:03.196339 26654 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 19:16:03.199695 26669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:03.201587 26654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 19:16:03.201814 26654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "d990c3e9c0ce4aec94ec2fac52a08112"
format_stamp: "Formatted at 2026-09-24 19:15:57 on dist-test-slave-0lgr"
server_key: "8a44059a36e2719a9886165890ebff41"
server_key_iv: "4f69affc7f6e909512171e5cbdec9758"
server_key_version: "encryptionkey@0"
I20260924 19:16:03.202934 26654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:03.249552 26654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:03.254545 26654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:03.254865 26654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:03.256019 26654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:03.257591 26654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:03.257726 26654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:03.257877 26654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:03.257975 26654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:03.263574 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:03.262535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:54925)
0924 19:16:03.263130 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:03.263283 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":263,"threads_started":1}
I20260924 19:16:03.296825 26654 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:39679
I20260924 19:16:03.296923 26782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:39679 every 8 connection(s)
I20260924 19:16:03.298511 26654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:16:03.303947 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:03.301900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:54365)
0924 19:16:03.302829 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:03.303052 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":409,"threads_started":1}
W20260924 19:16:03.304888 26783 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:42123 (0 consecutive failures): Network error: Failed to ping master at 127.11.18.126:42123: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
I20260924 19:16:03.305852 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:03.305499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:39593)
0924 19:16:03.305657 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:03.305735 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 19:16:03.306893 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:03.306409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:49375)
0924 19:16:03.306627 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:03.306739 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 19:16:03.307204 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 26654
W20260924 19:16:03.307204 26783 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:16:03.772904 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:03.771826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:52409)
0924 19:16:03.772495 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:03.772631 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":331,"threads_started":1}
I20260924 19:16:04.265707 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:04.264771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:53759)
0924 19:16:04.265392 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:04.265504 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":229,"threads_started":1}
I20260924 19:16:04.308688 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:04.307826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:41357)
0924 19:16:04.308332 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:04.308455 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260924 19:16:04.774871 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:04.774079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:46261)
0924 19:16:04.774576 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:04.774687 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":251,"threads_started":1}
I20260924 19:16:05.267503 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:05.266709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:38111)
0924 19:16:05.267236 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:05.267312 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":308,"threads_started":1}
I20260924 19:16:05.310611 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:05.309861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:45377)
0924 19:16:05.310344 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:05.310434 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260924 19:16:05.776989 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:05.776131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:42223)
0924 19:16:05.776712 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:05.776803 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":347,"threads_started":1}
I20260924 19:16:06.269484 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:06.268525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:56335)
0924 19:16:06.269155 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:06.269242 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":410,"threads_started":1}
I20260924 19:16:06.312475 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:06.311616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:35453)
0924 19:16:06.312088 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:06.312185 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":244,"threads_started":1}
I20260924 19:16:06.779173 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:06.778175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:39923)
0924 19:16:06.778867 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:06.778962 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":438,"threads_started":1}
I20260924 19:16:07.271451 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:07.270579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:55503)
0924 19:16:07.271084 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:07.271177 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":253,"threads_started":1}
I20260924 19:16:07.314342 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:07.313589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:40563)
0924 19:16:07.314076 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:07.314163 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
I20260924 19:16:07.781096 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:07.780249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:54831)
0924 19:16:07.780777 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:07.780858 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":279,"threads_started":1}
I20260924 19:16:08.273128 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:08.272362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:41883)
0924 19:16:08.272841 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:08.272947 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260924 19:16:08.316128 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:08.315387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:58227)
0924 19:16:08.315873 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:08.315956 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":241,"threads_started":1}
I20260924 19:16:08.783052 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:08.782279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:40087)
0924 19:16:08.782765 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:08.782885 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260924 19:16:09.275277 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:09.274268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:56801)
0924 19:16:09.274903 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:09.275021 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":257,"threads_started":1}
I20260924 19:16:09.318073 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:09.317254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:55899)
0924 19:16:09.317801 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:09.317898 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":315,"threads_started":1}
I20260924 19:16:09.785188 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:09.784295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:39189)
0924 19:16:09.784851 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:09.784957 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":247,"threads_started":1}
I20260924 19:16:10.277556 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:10.276624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:44677)
0924 19:16:10.277173 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:10.277267 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":316,"threads_started":1}
I20260924 19:16:10.319917 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:10.319086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:45119)
0924 19:16:10.319611 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:10.319698 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":244,"threads_started":1}
I20260924 19:16:10.787302 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:10.786455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:35373)
0924 19:16:10.787010 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:10.787112 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":306,"threads_started":1}
I20260924 19:16:11.279317 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:11.278569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:50833)
0924 19:16:11.279039 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:11.279136 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":226,"threads_started":1}
I20260924 19:16:11.322006 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:11.321043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:42833)
0924 19:16:11.321724 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:11.321831 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":386,"threads_started":1}
I20260924 19:16:11.789701 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:11.788501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:45093)
0924 19:16:11.789306 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:11.789444 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":529,"threads_started":1}
I20260924 19:16:12.281266 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:12.280503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:49807)
0924 19:16:12.281007 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:12.281098 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":285,"threads_started":1}
I20260924 19:16:12.323980 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:12.323045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:34919)
0924 19:16:12.323686 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:12.323790 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":352,"threads_started":1}
I20260924 19:16:12.791618 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:12.790807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:47981)
0924 19:16:12.791348 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:12.791437 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":321,"threads_started":1}
I20260924 19:16:13.283486 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:13.282587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:52949)
0924 19:16:13.283159 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:13.283270 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":315,"threads_started":1}
I20260924 19:16:13.325856 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:13.324984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:58691)
0924 19:16:13.325589 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:13.325679 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":263,"threads_started":1}
I20260924 19:16:13.793650 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:13.792725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:33633)
0924 19:16:13.793192 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:13.793293 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":232,"threads_started":1}
I20260924 19:16:14.285269 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:14.284540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:54659)
0924 19:16:14.285016 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:14.285103 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260924 19:16:14.327908 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:14.327062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:33683)
0924 19:16:14.327610 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:14.327711 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":310,"threads_started":1}
I20260924 19:16:14.795529 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:14.794690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:48107)
0924 19:16:14.795176 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:14.795309 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":270,"threads_started":1}
I20260924 19:16:15.287485 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:15.286436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:38731)
0924 19:16:15.286996 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:15.287094 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":304,"threads_started":1}
I20260924 19:16:15.329855 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:15.328980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:48615)
0924 19:16:15.329593 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:15.329676 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":397,"threads_started":1}
I20260924 19:16:15.797240 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:15.796468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:33767)
0924 19:16:15.796944 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:15.797051 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":225,"threads_started":1}
I20260924 19:16:16.289649 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:16.288540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:52071)
0924 19:16:16.289086 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:16.289189 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
I20260924 19:16:16.331851 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:16.330976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:42251)
0924 19:16:16.331513 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:16.331630 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":255,"threads_started":1}
I20260924 19:16:16.799059 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:16.798212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:36219)
0924 19:16:16.798741 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:16.798838 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":306,"threads_started":1}
I20260924 19:16:17.291836 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:17.290888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:47191)
0924 19:16:17.291510 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:17.291634 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":332,"threads_started":1}
I20260924 19:16:17.333612 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:17.332784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:39019)
0924 19:16:17.333248 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:17.333426 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":247,"threads_started":1}
I20260924 19:16:17.801424 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:17.800467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.66:49417)
0924 19:16:17.801022 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:17.801128 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":302,"threads_started":1}
I20260924 19:16:18.293906 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:18.293004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.65:39967)
0924 19:16:18.293623 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:18.293735 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":366,"threads_started":1}
I20260924 19:16:18.317380 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 26358
I20260924 19:16:18.335733 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:18.334851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42123 (local address 127.11.18.67:35799)
0924 19:16:18.335362 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:18.335459 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":280,"threads_started":1}
I20260924 19:16:18.338364 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:18.318716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.11.18.65:38773)
0924 19:16:18.319435 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:18.319451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:18.319494 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:16:18.321888 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:18.321898 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:18.322368 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:18.322685 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:18.324144 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:18.325520 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:18.327047 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:18.327877 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:18.328034 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:18.331298 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:18.331307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:18.331326 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:18.331369 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:18.334663 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:18.335592 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:18.335601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:18.335610 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:18.335837 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:18.336310 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:18.336315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:18.336318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:18.336908 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:18.337142 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:18.337704 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 19:16:18.338061 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":386,"threads_started":1}
I20260924 19:16:18.339920 26515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:18.424985 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 26358
I20260924 19:16:18.442479 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 26523
I20260924 19:16:18.461495 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:18.443523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.11.18.66:33769)
0924 19:16:18.444028 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:18.444042 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:18.444079 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:16:18.445636 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:18.445646 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:18.446071 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:18.446437 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:18.448318 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:18.449678 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:18.451022 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:18.451709 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:18.451909 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:18.454816 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:18.454824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:18.454840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:18.454881 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:18.457931 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:18.458657 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:18.458664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:18.458671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:18.458841 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:18.459234 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:18.459239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:18.459242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:18.459876 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:18.460207 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:18.460676 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 19:16:18.461132 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":195,"threads_started":1}
I20260924 19:16:18.462953 26646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:18.553022 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 26523
I20260924 19:16:18.572778 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 26654
I20260924 19:16:18.594107 26836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:18.574047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.11.18.67:39679)
0924 19:16:18.574687 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:18.574702 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:18.574744 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:16:18.576438 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:18.576450 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:18.576902 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:18.577238 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:18.578998 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:18.580296 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:18.581926 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:18.582673 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:18.582829 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:18.586109 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:18.586118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:18.586137 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:18.586185 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:18.589875 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:18.590918 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:18.590926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:18.590933 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:18.591136 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:18.591723 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:18.591727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:18.591729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:18.592346 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:18.592604 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:18.593142 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 19:16:18.593768 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":344,"threads_started":1}
I20260924 19:16:18.595688 26778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:18.683693 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 26654
2026-09-24T19:16:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25213 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:16:18 dist-test-slave-0lgr krb5kdc[26840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:16:18 dist-test-slave-0lgr krb5kdc[26840](info): set up 2 sockets
Sep 24 19:16:18 dist-test-slave-0lgr krb5kdc[26840](info): commencing operation
krb5kdc: starting...
W20260924 19:16:19.878271 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:16:20 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:16:20.143843 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41685
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:41685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:16:20.490000 26853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:20.490319 26853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:20.490466 26853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:20.498001 26853 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:16:20.498116 26853 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:16:20.498157 26853 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:16:20.498245 26853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:16:20.498327 26853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:20.498449 26853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:20.498533 26853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:16:20.498617 26853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:16:20.508934 26853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:41685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:20.510375 26853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:20.511825 26853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:20.525264 26859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:20.525664 26858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:20.525969 26861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:20.526252 26853 server_base.cc:1034] running on GCE node
I20260924 19:16:20.527223 26853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:20.528918 26853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:20.530224 26853 hybrid_clock.cc:648] HybridClock initialized: now 1790277380530136 us; error 61 us; skew 500 ppm
I20260924 19:16:20.530731 26853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:20 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277380, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:20.534716 26853 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:16:20.536437 26853 webserver.cc:460] Webserver started at http://127.11.18.126:41341/ using document root <none> and password file <none>
I20260924 19:16:20.537189 26853 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:20.537380 26853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:20.537712 26853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:16:20.540951 26853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "3926b46bbe3043378072f94aff49c6cb"
format_stamp: "Formatted at 2026-09-24 19:16:20 on dist-test-slave-0lgr"
server_key: "de62ab4948b81bd8268f162e3b86a57d"
server_key_iv: "cf488e459ff110f1c86d03671b870553"
server_key_version: "encryptionkey@0"
I20260924 19:16:20.541888 26853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "3926b46bbe3043378072f94aff49c6cb"
format_stamp: "Formatted at 2026-09-24 19:16:20 on dist-test-slave-0lgr"
server_key: "de62ab4948b81bd8268f162e3b86a57d"
server_key_iv: "cf488e459ff110f1c86d03671b870553"
server_key_version: "encryptionkey@0"
I20260924 19:16:20.548223 26853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:16:20.552934 26868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:20.554800 26853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:16:20.555039 26853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "3926b46bbe3043378072f94aff49c6cb"
format_stamp: "Formatted at 2026-09-24 19:16:20 on dist-test-slave-0lgr"
server_key: "de62ab4948b81bd8268f162e3b86a57d"
server_key_iv: "cf488e459ff110f1c86d03671b870553"
server_key_version: "encryptionkey@0"
I20260924 19:16:20.555303 26853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:20.575707 26853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:20.581976 26853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:20.582309 26853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:20.605950 26853 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:41685
I20260924 19:16:20.605993 26919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:41685 every 8 connection(s)
I20260924 19:16:20.607571 26853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:16:20.609989 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 26853
I20260924 19:16:20.610208 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:16:20.611052 11337 external_mini_cluster.cc:1428] Setting key f4488163629231f20ca53c0411ac8f57
I20260924 19:16:20.612509 26920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:16:20 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:16:20.627876 26920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Bootstrap starting.
I20260924 19:16:20.631337 26920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:20.632470 26920 log.cc:824] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:20.636369 26920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: No bootstrap required, opened a new log
I20260924 19:16:20.637802 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:20.613243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.11.18.126:41685)
0924 19:16:20.614120 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:20.614131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:20.614171 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:16:20.615774 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:20.615795 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:20.616219 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:20.616575 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:20.618566 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:20.620347 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:20.622323 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:20.623133 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:20.623277 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:20.628751 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:20.628762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:20.628780 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:20.628824 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:20.632419 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:20.633440 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:20.633450 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:20.633459 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:20.633734 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:20.634246 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:20.634250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:20.634252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:20.635051 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:20.635281 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:20.636046 (+ 765us) server_negotiation.cc:299] Negotiation successful
0924 19:16:20.636551 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":368,"threads_started":1}
I20260924 19:16:20.645919 26920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:16:20.646368 26920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:20.646674 26920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3926b46bbe3043378072f94aff49c6cb, State: Initialized, Role: FOLLOWER
I20260924 19:16:20.647621 26920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:16:20.647987 26920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:16:20.648161 26920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:16:20.648366 26920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:20.650895 26920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:16:20.651782 26920 leader_election.cc:302] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3926b46bbe3043378072f94aff49c6cb; no voters:
I20260924 19:16:20.652690 26920 leader_election.cc:288] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:16:20.652880 26925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:16:20.653632 26925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 1 LEADER]: Becoming Leader. State: Replica: 3926b46bbe3043378072f94aff49c6cb, State: Running, Role: LEADER
I20260924 19:16:20.654250 26925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:16:20.654976 26920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:16:20.658094 26926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3926b46bbe3043378072f94aff49c6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } } }
I20260924 19:16:20.658059 26927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3926b46bbe3043378072f94aff49c6cb. Latest consensus state: current_term: 1 leader_uuid: "3926b46bbe3043378072f94aff49c6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } } }
I20260924 19:16:20.658593 26926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: This master's current role is: LEADER
I20260924 19:16:20.658625 26927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: This master's current role is: LEADER
I20260924 19:16:20.662642 26933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:16:20.668526 26933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:16:20.677394 26933 catalog_manager.cc:1347] Generated new cluster ID: 96a21145d0a94672aea21098a6bb242e
I20260924 19:16:20.677524 26933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:16:20.705354 26933 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:16:20.706892 26933 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:16:20.719985 26933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Generated new TSK 0
I20260924 19:16:20.720890 26933 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:16:20.805624 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41685
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:16:21.155071 26946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:21.155493 26946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:21.155701 26946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:21.165813 26946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:21.165943 26946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:21.166231 26946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:16:21.177862 26946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:21.179517 26946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:21.181102 26946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:21.197343 26954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:21.197302 26951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:21.198058 26952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:21.198225 26946 server_base.cc:1034] running on GCE node
I20260924 19:16:21.198962 26946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:21.200145 26946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:21.201475 26946 hybrid_clock.cc:648] HybridClock initialized: now 1790277381201412 us; error 88 us; skew 500 ppm
I20260924 19:16:21.201879 26946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:21 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:21.206183 26946 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:16:21.208073 26946 webserver.cc:460] Webserver started at http://127.11.18.65:36283/ using document root <none> and password file <none>
I20260924 19:16:21.208890 26946 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:21.209065 26946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:21.209497 26946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:16:21.213007 26946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "38e44768b8934a4bb3d790647bf80838"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "cb99b8fe5721cd6c727d7cbbfce0d9c7"
server_key_iv: "4db058db33d20b0428c4ca7eac68494f"
server_key_version: "encryptionkey@0"
I20260924 19:16:21.213999 26946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "38e44768b8934a4bb3d790647bf80838"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "cb99b8fe5721cd6c727d7cbbfce0d9c7"
server_key_iv: "4db058db33d20b0428c4ca7eac68494f"
server_key_version: "encryptionkey@0"
I20260924 19:16:21.220932 26946 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260924 19:16:21.225642 26961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:21.227259 26946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 19:16:21.227478 26946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "38e44768b8934a4bb3d790647bf80838"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "cb99b8fe5721cd6c727d7cbbfce0d9c7"
server_key_iv: "4db058db33d20b0428c4ca7eac68494f"
server_key_version: "encryptionkey@0"
I20260924 19:16:21.227717 26946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:21.251523 26946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:21.256278 26946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:21.256588 26946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:21.257665 26946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:21.259095 26946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:21.259205 26946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:21.259335 26946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:21.259418 26946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:21.297962 26946 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:45805
I20260924 19:16:21.298125 27073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:45805 every 8 connection(s)
I20260924 19:16:21.299546 26946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:16:21.305115 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 26946
I20260924 19:16:21.305405 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:16:21.306155 11337 external_mini_cluster.cc:1428] Setting key e1b392d47d0be74658575691d6caf3ed
Sep 24 19:16:21 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:16:21.325743 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:21.302263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:49355 (local address 127.11.18.126:41685)
0924 19:16:21.302716 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:21.302721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:21.303986 (+ 1265us) server_negotiation.cc:407] Connection header received
0924 19:16:21.306588 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:21.306595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:21.306719 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:21.306866 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:21.309006 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:21.309971 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:21.313194 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:21.313844 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:21.313991 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:21.319184 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:21.319190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:21.319193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:21.319230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:21.322360 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:21.323190 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:21.323196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:21.323202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:21.323335 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:21.323808 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:21.323815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:21.323819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:21.324122 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:21.324288 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:21.325073 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 19:16:21.325372 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":251,"threads_started":1}
I20260924 19:16:21.326772 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:21.302637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:49355)
0924 19:16:21.303591 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:21.303663 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:21.306238 (+ 2575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:21.307175 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:21.307189 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:21.307869 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:16:21.308778 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:21.308809 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:21.310163 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:21.310172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:21.312984 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:21.313006 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:21.314143 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:21.314154 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:21.314555 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:21.315467 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:16:21.315506 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:16:21.318907 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:16:21.322542 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:21.322552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:21.322579 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:21.323050 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:21.323466 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:21.323476 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:21.323479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:21.323675 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:21.324276 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:16:21.324285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:16:21.324659 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 19:16:21.324962 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 19:16:21.325558 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":261,"threads_started":1}
I20260924 19:16:21.328734 27074 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41685
I20260924 19:16:21.329111 27074 heartbeater.cc:464] Registering TS with master...
I20260924 19:16:21.329958 27074 heartbeater.cc:511] Master 127.11.18.126:41685 requested a full tablet report, sending...
I20260924 19:16:21.332293 26885 ts_manager.cc:194] Registered new tserver with Master: 38e44768b8934a4bb3d790647bf80838 (127.11.18.65:45805)
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:16:21.333945 26885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:49355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:16:21.370612 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41685
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:16:21.714015 27080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:21.714376 27080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:21.714552 27080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:21.722143 27080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:21.722324 27080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:21.722520 27080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:16:21.732750 27080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:21.734225 27080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:21.735625 27080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:21.749859 27085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:21.749759 27086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:21.750972 27088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:21.751278 27080 server_base.cc:1034] running on GCE node
I20260924 19:16:21.751943 27080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:21.753051 27080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:21.754290 27080 hybrid_clock.cc:648] HybridClock initialized: now 1790277381754217 us; error 65 us; skew 500 ppm
I20260924 19:16:21.754721 27080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:21 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:21.758450 27080 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:16:21.760401 27080 webserver.cc:460] Webserver started at http://127.11.18.66:37485/ using document root <none> and password file <none>
I20260924 19:16:21.761183 27080 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:21.761332 27080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:21.761772 27080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:16:21.765288 27080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "5ee7f41ec93048299c414573d7704adc"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "453dea7a3bb629d55f7665e292bc1960"
server_key_iv: "9b5936b41cf0bc3dabdacc42d8a5476c"
server_key_version: "encryptionkey@0"
I20260924 19:16:21.766240 27080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "5ee7f41ec93048299c414573d7704adc"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "453dea7a3bb629d55f7665e292bc1960"
server_key_iv: "9b5936b41cf0bc3dabdacc42d8a5476c"
server_key_version: "encryptionkey@0"
I20260924 19:16:21.772296 27080 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 19:16:21.776533 27095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:21.778373 27080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:16:21.778538 27080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "5ee7f41ec93048299c414573d7704adc"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "453dea7a3bb629d55f7665e292bc1960"
server_key_iv: "9b5936b41cf0bc3dabdacc42d8a5476c"
server_key_version: "encryptionkey@0"
I20260924 19:16:21.778744 27080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:21.809993 27080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:21.815095 27080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:21.815397 27080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:21.816468 27080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:21.817991 27080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:21.818077 27080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:21.818173 27080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:21.818221 27080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:21.857059 27080 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:36319
I20260924 19:16:21.857146 27207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:36319 every 8 connection(s)
I20260924 19:16:21.858521 27080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:16:21.866312 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 27080
I20260924 19:16:21.866539 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:16:21.867105 11337 external_mini_cluster.cc:1428] Setting key 6f17c050119c03ff755c4fc8b896334a
Sep 24 19:16:21 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:16:21.887545 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:21.861720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:58651 (local address 127.11.18.126:41685)
0924 19:16:21.863265 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:21.863271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:21.864105 (+ 834us) server_negotiation.cc:407] Connection header received
0924 19:16:21.866325 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:21.866333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:21.866443 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:21.866596 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:21.868601 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:21.869825 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:21.871627 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:21.872538 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:21.872700 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:21.881370 (+ 8670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:21.881380 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:21.881394 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:21.881434 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:21.884274 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:21.885078 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:21.885082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:21.885085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:21.885217 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:21.885694 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:21.885700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:21.885703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:21.886006 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:21.886170 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:21.886912 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 19:16:21.887146 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1414,"thread_start_us":1301,"threads_started":1}
I20260924 19:16:21.888590 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:21.862237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:58651)
0924 19:16:21.863930 (+ 1693us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:21.863980 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:21.865978 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:21.866912 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:21.866921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:21.867621 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:16:21.868386 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:21.868413 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:21.870009 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:21.870013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:21.871449 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:21.871469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:21.872802 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:21.872809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:21.873050 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:21.877847 (+ 4797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:16:21.877890 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:16:21.880948 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:16:21.884434 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:21.884444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:21.884462 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:21.884927 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:21.885350 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:21.885357 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:21.885362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:21.885564 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:21.886142 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:16:21.886150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:16:21.886513 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 19:16:21.886805 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 19:16:21.887408 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":515,"threads_started":1}
I20260924 19:16:21.890369 27208 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41685
I20260924 19:16:21.890726 27208 heartbeater.cc:464] Registering TS with master...
I20260924 19:16:21.891513 27208 heartbeater.cc:511] Master 127.11.18.126:41685 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
I20260924 19:16:21.893198 26885 ts_manager.cc:194] Registered new tserver with Master: 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319)
I20260924 19:16:21.894145 26885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:58651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:16:21.930166 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41685
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:16:22.277424 27214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:22.277765 27214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:22.277930 27214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:22.285701 27214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:22.285872 27214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:22.286046 27214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:16:22.296391 27214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:22.297876 27214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:22.299191 27214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:22.314162 27222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:22.314160 27220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:22.314683 27219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:22.315140 27214 server_base.cc:1034] running on GCE node
I20260924 19:16:22.315861 27214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:22.317070 27214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:22.318305 27214 hybrid_clock.cc:648] HybridClock initialized: now 1790277382318255 us; error 42 us; skew 500 ppm
I20260924 19:16:22.318698 27214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:22 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:22.322683 27214 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:16:22.324390 27214 webserver.cc:460] Webserver started at http://127.11.18.67:37599/ using document root <none> and password file <none>
I20260924 19:16:22.325124 27214 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:22.325233 27214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:22.325582 27214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:16:22.328661 27214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "9522b4c78c7c41f0b336a75bd653aa80"
format_stamp: "Formatted at 2026-09-24 19:16:22 on dist-test-slave-0lgr"
server_key: "9ca3a2b0bbcb8f95b274d3cb47e501a8"
server_key_iv: "86c1ed8396d6f6aef25529deab69329a"
server_key_version: "encryptionkey@0"
I20260924 19:16:22.329545 27214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "9522b4c78c7c41f0b336a75bd653aa80"
format_stamp: "Formatted at 2026-09-24 19:16:22 on dist-test-slave-0lgr"
server_key: "9ca3a2b0bbcb8f95b274d3cb47e501a8"
server_key_iv: "86c1ed8396d6f6aef25529deab69329a"
server_key_version: "encryptionkey@0"
I20260924 19:16:22.335425 27214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:16:22.337594 27074 heartbeater.cc:503] Master 127.11.18.126:41685 was elected leader, sending a full tablet report...
I20260924 19:16:22.339960 27229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:22.341691 27214 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:16:22.341894 27214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "9522b4c78c7c41f0b336a75bd653aa80"
format_stamp: "Formatted at 2026-09-24 19:16:22 on dist-test-slave-0lgr"
server_key: "9ca3a2b0bbcb8f95b274d3cb47e501a8"
server_key_iv: "86c1ed8396d6f6aef25529deab69329a"
server_key_version: "encryptionkey@0"
I20260924 19:16:22.342126 27214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:22.367486 27214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:22.372254 27214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:22.372604 27214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:22.373728 27214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:22.375145 27214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:22.375232 27214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:22.375372 27214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:22.375423 27214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:22.410662 27214 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:46061
I20260924 19:16:22.410751 27341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:46061 every 8 connection(s)
I20260924 19:16:22.411969 27214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:16:22.415654 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 27214
I20260924 19:16:22.415876 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:16:22.416379 11337 external_mini_cluster.cc:1428] Setting key b689889a91e1a5bf985ef9e16dcf2b82
Sep 24 19:16:22 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:16:22.436172 27344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.414978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:47967 (local address 127.11.18.126:41685)
0924 19:16:22.415536 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.415543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.416878 (+ 1335us) server_negotiation.cc:407] Connection header received
0924 19:16:22.419521 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.419528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.419622 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.419729 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:22.421468 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.422388 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.424208 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.424685 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.424821 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.428751 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:22.428756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:22.428760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:22.428797 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:22.432578 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:22.433540 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:22.433549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:22.433558 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:22.433687 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:22.434180 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:22.434190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:22.434193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:22.434487 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:22.434670 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.435489 (+ 819us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.435736 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":272,"threads_started":1}
I20260924 19:16:22.437222 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.415596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:47967)
0924 19:16:22.416664 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:22.416723 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:22.418988 (+ 2265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:22.420060 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:22.420071 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:22.420637 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:16:22.421229 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.421247 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.422543 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.422548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.424017 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.424031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.424903 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.424911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.425129 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.425971 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:16:22.425995 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:16:22.428511 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:16:22.432736 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:22.432747 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:22.432766 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:22.433391 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:22.433814 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:22.433821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:22.433826 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:22.434043 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:22.434634 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:16:22.434644 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:16:22.435040 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 19:16:22.435371 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 19:16:22.435962 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":512,"threads_started":1}
I20260924 19:16:22.438916 27342 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41685
I20260924 19:16:22.439289 27342 heartbeater.cc:464] Registering TS with master...
I20260924 19:16:22.440171 27342 heartbeater.cc:511] Master 127.11.18.126:41685 requested a full tablet report, sending...
I20260924 19:16:22.441820 26885 ts_manager.cc:194] Registered new tserver with Master: 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061)
I20260924 19:16:22.442651 26885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:47967
I20260924 19:16:22.445488 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:16:22.465531 27344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.450393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.11.18.126:41685)
0924 19:16:22.450576 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.450581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.451048 (+ 467us) server_negotiation.cc:407] Connection header received
0924 19:16:22.451186 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.451190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.451271 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.451379 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:22.453034 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.453990 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.455699 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.456257 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.456392 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.459119 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:22.459124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:22.459128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:22.459195 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:22.462459 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:22.463431 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:22.463435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:22.463438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:22.463565 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:22.464000 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:22.464004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:22.464006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:22.464444 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:22.464653 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.464957 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.465139 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:16:22.470067 26885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:16:22.478307 26885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:16:22.504997 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.494044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.11.18.67:46061)
0924 19:16:22.494895 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.494907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.496413 (+ 1506us) server_negotiation.cc:407] Connection header received
0924 19:16:22.496520 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.496527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.496766 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.496987 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:22.498130 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.499569 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.502392 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.503810 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.503931 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.504179 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.504462 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.504633 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":497,"threads_started":1}
I20260924 19:16:22.505101 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.493545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46061 (local address 127.0.0.1:56134)
0924 19:16:22.496122 (+ 2577us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:22.496141 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:22.496280 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:22.497296 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:22.497300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:22.497350 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:22.497907 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.497924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.499721 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.499726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.502196 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.502231 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.504082 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.504089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.504274 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.504298 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:16:22.504400 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:16:22.504592 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2436,"mutex_wait_us":1,"thread_start_us":1337,"threads_started":1}
I20260924 19:16:22.505815 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.492526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:36319 (local address 127.0.0.1:51268)
0924 19:16:22.495245 (+ 2719us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:22.495284 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:22.495501 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:22.496164 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:22.496171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:22.496305 (+ 134us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:22.496823 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.496836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.499496 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.499503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.501908 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.501921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.503752 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.503759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.503983 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.504051 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 19:16:22.504199 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:16:22.505389 (+ 1190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2495,"thread_start_us":3301,"threads_started":1}
I20260924 19:16:22.506650 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.492742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.11.18.66:36319)
0924 19:16:22.493749 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.493761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.495401 (+ 1640us) server_negotiation.cc:407] Connection header received
0924 19:16:22.495734 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.495741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.495947 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.497468 (+ 1521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:22.497792 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.499344 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.502075 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.503541 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.503700 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.503915 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.506075 (+ 2160us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.506282 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":1608,"threads_started":1}
I20260924 19:16:22.506695 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.496609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.11.18.65:45805)
0924 19:16:22.497475 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.497485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.498264 (+ 779us) server_negotiation.cc:407] Connection header received
0924 19:16:22.498467 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.498474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.498696 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.498900 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:22.499771 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.501389 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.503533 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.505143 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.505307 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.505565 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.506194 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.506367 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":512,"threads_started":1}
I20260924 19:16:22.507827 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.497466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:45730)
0924 19:16:22.498130 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:22.498151 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:22.498297 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:22.498902 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:22.498908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:22.498934 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:22.499315 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.499327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.501549 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.501555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.503361 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.503373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.505714 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.505721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.505950 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.505992 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:16:22.507354 (+ 1362us) client_negotiation.cc:240] Negotiation successful
0924 19:16:22.507494 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":377,"threads_started":1}
I20260924 19:16:22.509800 27277 tablet_service.cc:1467] Processing CreateTablet for tablet 68ee919c9fde4d998b70129c9696304c (DEFAULT_TABLE table=test-table [id=04160068c0fa40e2b2d692f86ef38cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:22.510254 27143 tablet_service.cc:1467] Processing CreateTablet for tablet 68ee919c9fde4d998b70129c9696304c (DEFAULT_TABLE table=test-table [id=04160068c0fa40e2b2d692f86ef38cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:22.511600 27277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ee919c9fde4d998b70129c9696304c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:22.511700 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ee919c9fde4d998b70129c9696304c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:22.511590 27009 tablet_service.cc:1467] Processing CreateTablet for tablet 68ee919c9fde4d998b70129c9696304c (DEFAULT_TABLE table=test-table [id=04160068c0fa40e2b2d692f86ef38cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:22.513077 27009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ee919c9fde4d998b70129c9696304c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:22.526422 27361 tablet_bootstrap.cc:492] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Bootstrap starting.
I20260924 19:16:22.530176 27362 tablet_bootstrap.cc:492] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Bootstrap starting.
I20260924 19:16:22.530427 27361 tablet_bootstrap.cc:654] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:22.532226 27361 log.cc:824] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:22.534320 27363 tablet_bootstrap.cc:492] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Bootstrap starting.
I20260924 19:16:22.535610 27362 tablet_bootstrap.cc:654] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:22.536266 27361 tablet_bootstrap.cc:492] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: No bootstrap required, opened a new log
I20260924 19:16:22.536577 27361 ts_tablet_manager.cc:1397] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 19:16:22.537106 27362 log.cc:824] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:22.538507 27363 tablet_bootstrap.cc:654] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:22.539954 27363 log.cc:824] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:22.540863 27362 tablet_bootstrap.cc:492] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: No bootstrap required, opened a new log
I20260924 19:16:22.541186 27362 ts_tablet_manager.cc:1397] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 19:16:22.543507 27363 tablet_bootstrap.cc:492] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: No bootstrap required, opened a new log
I20260924 19:16:22.543831 27363 ts_tablet_manager.cc:1397] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 19:16:22.546506 27361 raft_consensus.cc:348] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.547045 27361 raft_consensus.cc:374] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:22.547158 27361 raft_consensus.cc:729] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9522b4c78c7c41f0b336a75bd653aa80, State: Initialized, Role: FOLLOWER
I20260924 19:16:22.547902 27361 consensus_queue.cc:260] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.549291 27342 heartbeater.cc:503] Master 127.11.18.126:41685 was elected leader, sending a full tablet report...
I20260924 19:16:22.550155 27361 ts_tablet_manager.cc:1428] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260924 19:16:22.554004 27362 raft_consensus.cc:348] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.554545 27362 raft_consensus.cc:374] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:22.554654 27362 raft_consensus.cc:729] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee7f41ec93048299c414573d7704adc, State: Initialized, Role: FOLLOWER
I20260924 19:16:22.555414 27362 consensus_queue.cc:260] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.556722 27208 heartbeater.cc:503] Master 127.11.18.126:41685 was elected leader, sending a full tablet report...
I20260924 19:16:22.556444 27363 raft_consensus.cc:348] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.556908 27363 raft_consensus.cc:374] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:22.557039 27363 raft_consensus.cc:729] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e44768b8934a4bb3d790647bf80838, State: Initialized, Role: FOLLOWER
I20260924 19:16:22.557587 27362 ts_tablet_manager.cc:1428] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Time spent starting tablet: real 0.016s user 0.019s sys 0.000s
I20260924 19:16:22.557773 27363 consensus_queue.cc:260] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.560240 27363 ts_tablet_manager.cc:1428] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Time spent starting tablet: real 0.016s user 0.007s sys 0.010s
W20260924 19:16:22.611110 27209 tablet.cc:2369] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:16:22.663758 27343 tablet.cc:2369] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:16:22.751374 27368 raft_consensus.cc:482] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:16:22.751780 27368 raft_consensus.cc:504] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.753723 27368 leader_election.cc:288] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38e44768b8934a4bb3d790647bf80838 (127.11.18.65:45805), 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061)
I20260924 19:16:22.763372 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.754151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:53107 (local address 127.11.18.65:45805)
0924 19:16:22.754378 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.754385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.755474 (+ 1089us) server_negotiation.cc:407] Connection header received
0924 19:16:22.755676 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.755681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.755797 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.755899 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:22.756559 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.757641 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.760599 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.761937 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.762121 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.762199 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.762941 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.763094 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:16:22.763465 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.754415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:50433 (local address 127.11.18.67:46061)
0924 19:16:22.754694 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.754704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.755083 (+ 379us) server_negotiation.cc:407] Connection header received
0924 19:16:22.755331 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.755335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.755410 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.755524 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:22.756181 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.757410 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.760604 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.761976 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.762129 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.762199 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.762941 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.763094 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:16:22.763541 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.754106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46061 (local address 127.11.18.66:50433)
0924 19:16:22.754957 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:22.754973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:22.755171 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:22.755570 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:22.755577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:22.755608 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:22.756025 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.756040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.757581 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.757585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.760418 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.760433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.762266 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.762271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.762592 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.762738 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 19:16:22.762900 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 19:16:22.763023 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":245,"thread_start_us":485,"threads_started":1}
I20260924 19:16:22.764000 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.754150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.11.18.66:53107)
0924 19:16:22.755357 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:22.755373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:22.755525 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:22.755986 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:22.755991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:22.756022 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:22.756392 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.756401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.757787 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.757798 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.760415 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:22.760433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.762260 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.762268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:22.762591 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.762749 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 19:16:22.762850 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:16:22.763007 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"mutex_wait_us":158,"thread_start_us":833,"threads_started":1}
I20260924 19:16:22.764875 27297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ee919c9fde4d998b70129c9696304c" candidate_uuid: "5ee7f41ec93048299c414573d7704adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9522b4c78c7c41f0b336a75bd653aa80" is_pre_election: true
I20260924 19:16:22.764995 27029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ee919c9fde4d998b70129c9696304c" candidate_uuid: "5ee7f41ec93048299c414573d7704adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e44768b8934a4bb3d790647bf80838" is_pre_election: true
I20260924 19:16:22.765588 27297 raft_consensus.cc:2454] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee7f41ec93048299c414573d7704adc in term 0.
I20260924 19:16:22.765588 27029 raft_consensus.cc:2454] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee7f41ec93048299c414573d7704adc in term 0.
I20260924 19:16:22.766400 27097 leader_election.cc:302] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e44768b8934a4bb3d790647bf80838, 5ee7f41ec93048299c414573d7704adc; no voters:
I20260924 19:16:22.766831 27368 raft_consensus.cc:2781] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:16:22.767005 27368 raft_consensus.cc:482] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:16:22.767122 27368 raft_consensus.cc:3037] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:22.769568 27368 raft_consensus.cc:504] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.770285 27368 leader_election.cc:288] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [CANDIDATE]: Term 1 election: Requested vote from peers 38e44768b8934a4bb3d790647bf80838 (127.11.18.65:45805), 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061)
I20260924 19:16:22.770888 27297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ee919c9fde4d998b70129c9696304c" candidate_uuid: "5ee7f41ec93048299c414573d7704adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9522b4c78c7c41f0b336a75bd653aa80"
I20260924 19:16:22.770908 27029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ee919c9fde4d998b70129c9696304c" candidate_uuid: "5ee7f41ec93048299c414573d7704adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e44768b8934a4bb3d790647bf80838"
I20260924 19:16:22.771116 27297 raft_consensus.cc:3037] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:22.771132 27029 raft_consensus.cc:3037] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:22.773563 27297 raft_consensus.cc:2454] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee7f41ec93048299c414573d7704adc in term 1.
I20260924 19:16:22.773730 27029 raft_consensus.cc:2454] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee7f41ec93048299c414573d7704adc in term 1.
I20260924 19:16:22.774035 27096 leader_election.cc:302] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee7f41ec93048299c414573d7704adc, 9522b4c78c7c41f0b336a75bd653aa80; no voters:
I20260924 19:16:22.774369 27368 raft_consensus.cc:2781] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:16:22.774892 27368 raft_consensus.cc:686] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 1 LEADER]: Becoming Leader. State: Replica: 5ee7f41ec93048299c414573d7704adc, State: Running, Role: LEADER
I20260924 19:16:22.775411 27368 consensus_queue.cc:237] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } }
I20260924 19:16:22.780236 26885 catalog_manager.cc:5495] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee7f41ec93048299c414573d7704adc (127.11.18.66). New cstate: current_term: 1 leader_uuid: "5ee7f41ec93048299c414573d7704adc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } health_report { overall_health: UNKNOWN } } }
W20260924 19:16:22.802448 27075 tablet.cc:2369] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:16:22.805191 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.798993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.11.18.66:36319)
0924 19:16:22.799207 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.799211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.799231 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:16:22.799511 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.799517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.799602 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.799753 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:16:22.800170 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.801010 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.802793 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.803243 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.803409 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.803947 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:16:22.804506 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:16:22.804701 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.804797 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.804893 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:16:22.814076 27297 raft_consensus.cc:1261] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 FOLLOWER]: Refusing update from remote peer 5ee7f41ec93048299c414573d7704adc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:16:22.814105 27029 raft_consensus.cc:1261] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Refusing update from remote peer 5ee7f41ec93048299c414573d7704adc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:16:22.815198 27372 consensus_queue.cc:1035] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:16:22.815649 27368 consensus_queue.cc:1035] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:16:22.833346 27375 mvcc.cc:205] Tried to move back new op lower bound from 7332976159995752448 to 7332976159851929600. Current Snapshot: MvccSnapshot[applied={T|T < 7332976159995752448}]
I20260924 19:16:22.839738 27376 mvcc.cc:205] Tried to move back new op lower bound from 7332976159995752448 to 7332976159851929600. Current Snapshot: MvccSnapshot[applied={T|T < 7332976159995752448}]
I20260924 19:16:22.862028 26885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:16:22.862319 26885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:16:22.868180 26885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Sending DeleteTablet for 3 replicas of tablet 68ee919c9fde4d998b70129c9696304c
I20260924 19:16:22.869688 27143 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ee919c9fde4d998b70129c9696304c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:22 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:51268
I20260924 19:16:22.869745 27277 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ee919c9fde4d998b70129c9696304c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:22 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:56134
I20260924 19:16:22.870656 27009 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ee919c9fde4d998b70129c9696304c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:22 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:45730
I20260924 19:16:22.871047 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 26946
I20260924 19:16:22.886508 27388 tablet_replica.cc:330] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: stopping tablet replica
I20260924 19:16:22.887153 27388 raft_consensus.cc:2229] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:16:22.887840 27388 raft_consensus.cc:2258] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:16:22.895162 27387 tablet_replica.cc:330] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: stopping tablet replica
I20260924 19:16:22.895767 27387 raft_consensus.cc:2229] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:16:22.896440 27387 raft_consensus.cc:2258] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:16:22.898689 27386 tablet_replica.cc:330] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: stopping tablet replica
I20260924 19:16:22.899165 27386 raft_consensus.cc:2229] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:16:22.899319 27388 ts_tablet_manager.cc:1905] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:16:22.899999 27386 raft_consensus.cc:2258] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:16:22.902079 27387 ts_tablet_manager.cc:1905] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:16:22.908231 27388 ts_tablet_manager.cc:1918] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:16:22.908483 27388 log.cc:1196] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/68ee919c9fde4d998b70129c9696304c
I20260924 19:16:22.909111 27388 ts_tablet_manager.cc:1939] T 68ee919c9fde4d998b70129c9696304c P 38e44768b8934a4bb3d790647bf80838: Deleting consensus metadata
Sep 24 19:16:22 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:16:22.911335 26870 catalog_manager.cc:4841] TS 38e44768b8934a4bb3d790647bf80838 (127.11.18.65:45805): tablet 68ee919c9fde4d998b70129c9696304c (table test-table [id=04160068c0fa40e2b2d692f86ef38cdf]) successfully deleted
I20260924 19:16:22.913357 27386 ts_tablet_manager.cc:1905] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:16:22.918493 27387 ts_tablet_manager.cc:1918] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:16:22.918680 27387 log.cc:1196] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/68ee919c9fde4d998b70129c9696304c
I20260924 19:16:22.919260 27387 ts_tablet_manager.cc:1939] T 68ee919c9fde4d998b70129c9696304c P 9522b4c78c7c41f0b336a75bd653aa80: Deleting consensus metadata
I20260924 19:16:22.921485 26869 catalog_manager.cc:4841] TS 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061): tablet 68ee919c9fde4d998b70129c9696304c (table test-table [id=04160068c0fa40e2b2d692f86ef38cdf]) successfully deleted
I20260924 19:16:22.921949 27386 ts_tablet_manager.cc:1918] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:16:22.922139 27386 log.cc:1196] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/68ee919c9fde4d998b70129c9696304c
I20260924 19:16:22.922278 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:22.872628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45738 (local address 127.11.18.65:45805)
0924 19:16:22.872877 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:22.872886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:22.897591 (+ 24705us) server_negotiation.cc:407] Connection header received
0924 19:16:22.898197 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:22.898206 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:22.898305 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:22.898435 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:22.901072 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.902366 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.904753 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:22.905550 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:22.905655 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:22.912935 (+ 7280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:22.912951 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:22.912970 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:22.913023 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:22.917631 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:22.918973 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:22.918986 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:22.918994 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:22.919195 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:22.919817 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:22.919826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:22.919833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:22.920349 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:22.920649 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:22.921383 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 19:16:22.921725 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 19:16:22.922761 27386 ts_tablet_manager.cc:1939] T 68ee919c9fde4d998b70129c9696304c P 5ee7f41ec93048299c414573d7704adc: Deleting consensus metadata
I20260924 19:16:22.923502 27069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:22.924757 26870 catalog_manager.cc:4841] TS 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319): tablet 68ee919c9fde4d998b70129c9696304c (table test-table [id=04160068c0fa40e2b2d692f86ef38cdf]) successfully deleted
I20260924 19:16:23.023324 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 26946
I20260924 19:16:23.041949 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 27080
Sep 24 19:16:23 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:16:23.062165 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:23.042857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.11.18.66:36319)
0924 19:16:23.043049 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:23.043054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:23.043073 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:16:23.043354 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:23.043359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:23.043458 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:23.043556 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:23.045430 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:23.046271 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:23.048453 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:23.048949 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:23.049074 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:23.054479 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:23.054496 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:23.054519 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:23.054569 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:23.058601 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:23.059679 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:23.059691 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:23.059694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:23.059907 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:23.060370 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:23.060378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:23.060382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:23.060787 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:23.061201 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:23.061504 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:16:23.061756 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:16:23.062964 27203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:23.179522 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 27080
I20260924 19:16:23.198658 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 27214
Sep 24 19:16:23 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:16:23.215445 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:23.199520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.11.18.67:46061)
0924 19:16:23.199725 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:23.199731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:23.199752 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:16:23.199989 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:23.199994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:23.200063 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:23.200180 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:23.201518 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:23.202246 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:23.203540 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:23.203951 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:23.204090 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:23.209290 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:23.209458 (+ 168us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:23.209486 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:23.209531 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:23.212562 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:23.213447 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:23.213455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:23.213457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:23.213614 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:23.214013 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:23.214016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:23.214018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:23.214404 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:23.214632 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:23.214948 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 19:16:23.215123 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:16:23.216218 27337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:23.310437 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 27214
I20260924 19:16:23.328558 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 26853
I20260924 19:16:23.329969 26915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:23.445761 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 26853
I20260924 19:16:43.468420 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41685
--webserver_interface=127.11.18.126
--webserver_port=41341
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:41685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:16:43.803022 27394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:43.803323 27394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:43.803484 27394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:43.810905 27394 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:16:43.811058 27394 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:16:43.811136 27394 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:16:43.811219 27394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:16:43.811277 27394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:43.811350 27394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:43.811398 27394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:16:43.811446 27394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:16:43.821640 27394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:41685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=41341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:43.822935 27394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:43.824328 27394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:43.849392 27401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:43.849545 27403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:43.849567 27400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:43.850165 27394 server_base.cc:1034] running on GCE node
I20260924 19:16:43.851207 27394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:43.853040 27394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:43.854422 27394 hybrid_clock.cc:648] HybridClock initialized: now 1790277403854208 us; error 88 us; skew 500 ppm
I20260924 19:16:43.854908 27394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:43 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277403, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:43.862916 27394 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:16:43.864884 27394 webserver.cc:460] Webserver started at http://127.11.18.126:41341/ using document root <none> and password file <none>
I20260924 19:16:43.865825 27394 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:43.865969 27394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:43.872421 27394 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 19:16:43.875658 27410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:43.877134 27394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:16:43.877385 27394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "3926b46bbe3043378072f94aff49c6cb"
format_stamp: "Formatted at 2026-09-24 19:16:20 on dist-test-slave-0lgr"
server_key: "de62ab4948b81bd8268f162e3b86a57d"
server_key_iv: "cf488e459ff110f1c86d03671b870553"
server_key_version: "encryptionkey@0"
I20260924 19:16:43.878494 27394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:43.895380 27394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:43.900318 27394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:43.900699 27394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:43.922384 27394 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:41685
I20260924 19:16:43.922400 27461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:41685 every 8 connection(s)
I20260924 19:16:43.924306 27394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:16:43.933473 27462 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:16:43.934094 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 27394
I20260924 19:16:43.935365 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:45805
--local_ip_for_outbound_sockets=127.11.18.65
--tserver_master_addrs=127.11.18.126:41685
--webserver_port=36283
--webserver_interface=127.11.18.65
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:16:43.937426 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Bootstrap starting.
I20260924 19:16:43.968546 27462 log.cc:824] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:43.985157 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:16:43.985932 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Bootstrap complete.
I20260924 19:16:44.000166 27462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:16:44.001600 27462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3926b46bbe3043378072f94aff49c6cb, State: Initialized, Role: FOLLOWER
I20260924 19:16:44.002341 27462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:16:44.002667 27462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:16:44.002827 27462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:16:44.003042 27462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:16:44.007597 27462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:16:44.008208 27462 leader_election.cc:302] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3926b46bbe3043378072f94aff49c6cb; no voters:
I20260924 19:16:44.008978 27462 leader_election.cc:288] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:16:44.009140 27466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:16:44.010605 27466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 2 LEADER]: Becoming Leader. State: Replica: 3926b46bbe3043378072f94aff49c6cb, State: Running, Role: LEADER
I20260924 19:16:44.011209 27466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:16:44.011785 27462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:16:44.013973 27467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3926b46bbe3043378072f94aff49c6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } } }
I20260924 19:16:44.014345 27467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: This master's current role is: LEADER
I20260924 19:16:44.014015 27468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3926b46bbe3043378072f94aff49c6cb. Latest consensus state: current_term: 2 leader_uuid: "3926b46bbe3043378072f94aff49c6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } } }
I20260924 19:16:44.014496 27468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: This master's current role is: LEADER
I20260924 19:16:44.026965 27471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:16:44.040004 27471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:16:44.041900 27471 catalog_manager.cc:1259] Loaded cluster ID: 96a21145d0a94672aea21098a6bb242e
I20260924 19:16:44.042428 27471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:16:44.048084 27471 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:16:44.050540 27471 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Loaded TSK: 0
I20260924 19:16:44.084705 27471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Generated new TSK 1
I20260924 19:16:44.085139 27471 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:16:44.346546 27464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:44.346894 27464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:44.347070 27464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:44.354761 27464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:44.354878 27464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:44.355049 27464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:16:44.365998 27464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:45805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=36283
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:44.367393 27464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:44.368767 27464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:44.382576 27490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:44.382642 27492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:44.383561 27489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:44.384948 27464 server_base.cc:1034] running on GCE node
I20260924 19:16:44.385793 27464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:44.387105 27464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:44.388363 27464 hybrid_clock.cc:648] HybridClock initialized: now 1790277404388302 us; error 51 us; skew 500 ppm
I20260924 19:16:44.388746 27464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:44 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:44.392903 27464 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:16:44.394814 27464 webserver.cc:460] Webserver started at http://127.11.18.65:36283/ using document root <none> and password file <none>
I20260924 19:16:44.395740 27464 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:44.395906 27464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:44.403426 27464 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:16:44.407420 27499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:44.409313 27464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 19:16:44.409602 27464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "38e44768b8934a4bb3d790647bf80838"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "cb99b8fe5721cd6c727d7cbbfce0d9c7"
server_key_iv: "4db058db33d20b0428c4ca7eac68494f"
server_key_version: "encryptionkey@0"
I20260924 19:16:44.411172 27464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:44.469532 27464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:44.474489 27464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:44.474736 27464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:44.475773 27464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:44.477134 27464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:44.477229 27464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:44.477303 27464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:44.477413 27464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 19:16:44.512480 27464 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:45805
I20260924 19:16:44.512583 27611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:45805 every 8 connection(s)
I20260924 19:16:44.514110 27464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:16:44.518106 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 27464
I20260924 19:16:44.518821 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:36319
--local_ip_for_outbound_sockets=127.11.18.66
--tserver_master_addrs=127.11.18.126:41685
--webserver_port=37485
--webserver_interface=127.11.18.66
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:16:44 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:16:44.543170 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:44.517459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:41467)
0924 19:16:44.518335 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:44.518379 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:44.520254 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:44.521901 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:44.521908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:44.522437 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:16:44.523018 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:44.523036 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:44.525448 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:44.525456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:44.527045 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:44.527058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:44.528277 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:44.528286 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:44.528629 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:44.529429 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:16:44.529463 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:16:44.532599 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:16:44.538148 (+ 5549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:44.538156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:44.538171 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:44.538854 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:44.539506 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:44.539510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:44.539512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:44.539723 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:44.540893 (+ 1170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:16:44.540900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:16:44.541235 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 19:16:44.541625 (+ 390us) client_negotiation.cc:240] Negotiation successful
0924 19:16:44.542068 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":373,"threads_started":1}
I20260924 19:16:44.543692 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:44.517066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:41467 (local address 127.11.18.126:41685)
0924 19:16:44.517730 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:44.517746 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:44.518530 (+ 784us) server_negotiation.cc:407] Connection header received
0924 19:16:44.520835 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:44.520854 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:44.521368 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:44.521773 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:44.523236 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:44.525240 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:44.527207 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:44.528000 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:44.528169 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:44.533019 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:44.533033 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:44.533056 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:44.533115 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:44.537932 (+ 4817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:44.539046 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:44.539060 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:44.539070 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:44.539320 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:44.539856 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:44.539864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:44.539870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:44.540635 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:44.540868 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:44.541909 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0924 19:16:44.542408 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
I20260924 19:16:44.546804 27612 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41685
I20260924 19:16:44.547194 27612 heartbeater.cc:464] Registering TS with master...
I20260924 19:16:44.548314 27612 heartbeater.cc:511] Master 127.11.18.126:41685 requested a full tablet report, sending...
I20260924 19:16:44.551127 27427 ts_manager.cc:194] Registered new tserver with Master: 38e44768b8934a4bb3d790647bf80838 (127.11.18.65:45805)
I20260924 19:16:44.552726 27427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:41467
W20260924 19:16:44.874740 27616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:44.875056 27616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:44.875221 27616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:44.883611 27616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:44.883741 27616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:44.883993 27616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:16:44.894683 27616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:36319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=37485
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:44.895990 27616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:44.897511 27616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:44.912191 27621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:44.912099 27622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:44.912329 27624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:44.913040 27616 server_base.cc:1034] running on GCE node
I20260924 19:16:44.913866 27616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:44.914942 27616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:44.916217 27616 hybrid_clock.cc:648] HybridClock initialized: now 1790277404916149 us; error 59 us; skew 500 ppm
I20260924 19:16:44.916708 27616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:44 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:44.920403 27616 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:16:44.922501 27616 webserver.cc:460] Webserver started at http://127.11.18.66:37485/ using document root <none> and password file <none>
I20260924 19:16:44.923440 27616 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:44.923611 27616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:44.931578 27616 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:16:44.935410 27631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:44.937548 27616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:16:44.937758 27616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "5ee7f41ec93048299c414573d7704adc"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "453dea7a3bb629d55f7665e292bc1960"
server_key_iv: "9b5936b41cf0bc3dabdacc42d8a5476c"
server_key_version: "encryptionkey@0"
I20260924 19:16:44.939273 27616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:44.955912 27616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:44.961676 27616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:44.961997 27616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:44.963198 27616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:44.964772 27616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:44.964910 27616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:44.965037 27616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:44.965126 27616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:44.999607 27616 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:36319
I20260924 19:16:44.999698 27743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:36319 every 8 connection(s)
I20260924 19:16:45.001080 27616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:16:45.008457 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 27616
I20260924 19:16:45.009251 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:46061
--local_ip_for_outbound_sockets=127.11.18.67
--tserver_master_addrs=127.11.18.126:41685
--webserver_port=37599
--webserver_interface=127.11.18.67
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:16:45 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:16:45.024884 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.003970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:45351 (local address 127.11.18.126:41685)
0924 19:16:45.004173 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.004178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.005378 (+ 1200us) server_negotiation.cc:407] Connection header received
0924 19:16:45.007475 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.007481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.007607 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.007730 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:45.009490 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.010353 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.012092 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.012701 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.012849 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.017871 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:45.017877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:45.017880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:45.017917 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:45.021001 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.022176 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.022180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.022183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.022373 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.022916 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.022919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.022921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.023255 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:45.023377 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.024418 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.024615 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:16:45.026602 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.004386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:45351)
0924 19:16:45.005161 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:45.005218 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:45.007140 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:45.007996 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:45.008005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:45.008556 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:16:45.009247 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.009266 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.010530 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.010536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.011926 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.011940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.012971 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.012978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.013254 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.014205 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:16:45.014236 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:16:45.017574 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:16:45.021180 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:45.021195 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:45.021229 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:45.022005 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:45.022504 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:45.022513 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:45.022520 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:45.022786 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:45.023406 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:16:45.023419 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:16:45.023966 (+ 547us) client_negotiation.cc:769] Sending connection context
0924 19:16:45.024302 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 19:16:45.024972 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":282,"threads_started":1}
I20260924 19:16:45.028415 27744 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41685
I20260924 19:16:45.028793 27744 heartbeater.cc:464] Registering TS with master...
I20260924 19:16:45.029657 27744 heartbeater.cc:511] Master 127.11.18.126:41685 requested a full tablet report, sending...
I20260924 19:16:45.031189 27427 ts_manager.cc:194] Registered new tserver with Master: 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319)
I20260924 19:16:45.032018 27427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:45351
W20260924 19:16:45.356079 27747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:16:45.356437 27747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:16:45.356635 27747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:16:45.364157 27747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:16:45.364329 27747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:16:45.364522 27747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:16:45.374881 27747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:46061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=37599
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:16:45.376371 27747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:16:45.377738 27747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:16:45.393990 27752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:45.394029 27753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:16:45.393949 27755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:16:45.395143 27747 server_base.cc:1034] running on GCE node
I20260924 19:16:45.395751 27747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:16:45.396864 27747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:16:45.398124 27747 hybrid_clock.cc:648] HybridClock initialized: now 1790277405398053 us; error 60 us; skew 500 ppm
I20260924 19:16:45.398597 27747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:16:45 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277405, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:16:45.402213 27747 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:16:45.403939 27747 webserver.cc:460] Webserver started at http://127.11.18.67:37599/ using document root <none> and password file <none>
I20260924 19:16:45.404654 27747 fs_manager.cc:359] Metadata directory not provided
I20260924 19:16:45.404784 27747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:16:45.411034 27747 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:16:45.414351 27762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:45.415854 27747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:16:45.416055 27747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "9522b4c78c7c41f0b336a75bd653aa80"
format_stamp: "Formatted at 2026-09-24 19:16:22 on dist-test-slave-0lgr"
server_key: "9ca3a2b0bbcb8f95b274d3cb47e501a8"
server_key_iv: "86c1ed8396d6f6aef25529deab69329a"
server_key_version: "encryptionkey@0"
I20260924 19:16:45.417264 27747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:16:45.448477 27747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:16:45.453543 27747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:16:45.453840 27747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:16:45.454869 27747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:16:45.456310 27747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:16:45.456435 27747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:45.456565 27747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:16:45.456652 27747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:16:45.492132 27747 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:46061
I20260924 19:16:45.492180 27874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:46061 every 8 connection(s)
I20260924 19:16:45.493611 27747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:16:45.498641 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 27747
Sep 24 19:16:45 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277405, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:16:45.517436 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.496260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:49517 (local address 127.11.18.126:41685)
0924 19:16:45.496492 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.496498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.497897 (+ 1399us) server_negotiation.cc:407] Connection header received
0924 19:16:45.500213 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.500221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.500342 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.500421 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:45.502411 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.503182 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.504828 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.505371 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.505490 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.509624 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:45.509635 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:45.509641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:45.509691 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:45.513531 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.514574 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.514579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.514583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.514896 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.515422 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.515428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.515432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.515705 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:45.515864 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.516724 (+ 860us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.516972 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":76}
I20260924 19:16:45.517815 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.499786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.11.18.126:41685)
0924 19:16:45.500461 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.500468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.500486 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:16:45.500531 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.500537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.500604 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.500705 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:45.502358 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.503287 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.504724 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.505251 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.505403 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.508680 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:45.508687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:45.508692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:45.508735 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:45.512514 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.513642 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.513651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.513658 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.513829 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.514377 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.514382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.514386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.514803 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:45.515083 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.516399 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.516663 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":445,"threads_started":1}
I20260924 19:16:45.519363 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.496780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:49517)
0924 19:16:45.497709 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:45.497756 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:45.499791 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:45.500971 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:45.500980 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:45.501586 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:16:45.502187 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.502205 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.503370 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.503378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.504684 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.504701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.505622 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.505629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.505837 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.506588 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:16:45.506611 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:16:45.509297 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:16:45.513723 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:45.513738 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:45.513751 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:45.514437 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:45.515051 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:16:45.515058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:16:45.515062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:16:45.515280 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:16:45.515884 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:16:45.515895 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:16:45.516324 (+ 429us) client_negotiation.cc:769] Sending connection context
0924 19:16:45.516628 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 19:16:45.517384 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":300,"threads_started":1}
I20260924 19:16:45.521080 27875 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41685
I20260924 19:16:45.521538 27875 heartbeater.cc:464] Registering TS with master...
I20260924 19:16:45.522343 27875 heartbeater.cc:511] Master 127.11.18.126:41685 requested a full tablet report, sending...
I20260924 19:16:45.523927 27427 ts_manager.cc:194] Registered new tserver with Master: 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061)
I20260924 19:16:45.524718 27427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:49517
I20260924 19:16:45.534574 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:16:45.542173 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.535692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.11.18.126:41685)
0924 19:16:45.535933 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.535943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.536607 (+ 664us) server_negotiation.cc:407] Connection header received
0924 19:16:45.536725 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.536728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.536875 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.537004 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:16:45.537471 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.538190 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.539679 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.540102 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.540324 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.540934 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:16:45.541154 (+ 220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 19:16:45.541719 (+ 565us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53026: token expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 19:16:45.542459 27879 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53026: token expired
W20260924 19:16:45.543040 27346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.18.126:41685)
I20260924 19:16:45.557070 27612 heartbeater.cc:503] Master 127.11.18.126:41685 was elected leader, sending a full tablet report...
I20260924 19:16:45.573894 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.543920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.11.18.126:41685)
0924 19:16:45.544146 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.544152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.544172 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:16:45.544228 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.544233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.544312 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.544421 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:45.548414 (+ 3993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.549857 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.555529 (+ 5672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.556378 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.556564 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.563284 (+ 6720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:45.563294 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:45.563304 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:45.563632 (+ 328us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:45.569418 (+ 5786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.570705 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.570713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.570717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.570945 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.571431 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.571439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.571445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.572034 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:45.572947 (+ 913us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.573108 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.573418 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 19:16:45.580026 27427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:16:45.591675 27427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:16:45.621567 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.607969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.11.18.66:36319)
0924 19:16:45.609104 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.609116 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.609708 (+ 592us) server_negotiation.cc:407] Connection header received
0924 19:16:45.609922 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.609929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.610149 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.610377 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:45.611641 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.613497 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.617168 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.619437 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.619633 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.619851 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.620774 (+ 923us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.621036 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":795,"threads_started":1}
I20260924 19:16:45.622716 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.606972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46061 (local address 127.0.0.1:35632)
0924 19:16:45.608143 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:45.608213 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:45.608431 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:45.610148 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:45.610178 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:45.610235 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:45.610869 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.610890 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.614499 (+ 3609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.614509 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.618146 (+ 3637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.618189 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.621298 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.621308 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.621617 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.621672 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 19:16:45.621750 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:16:45.621890 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":30,"thread_start_us":607,"threads_started":1}
I20260924 19:16:45.622784 27886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.607267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:36319 (local address 127.0.0.1:40520)
0924 19:16:45.609575 (+ 2308us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:45.609598 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:45.609720 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:45.610694 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:45.610703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:45.610743 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:45.611392 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.611419 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.613736 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.613745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.616794 (+ 3049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.616817 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.620091 (+ 3274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.620098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.620316 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.620379 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 19:16:45.620517 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 19:16:45.621991 (+ 1474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2185,"mutex_wait_us":8,"thread_start_us":2012,"threads_started":1}
I20260924 19:16:45.625123 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.607409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.11.18.67:46061)
0924 19:16:45.609353 (+ 1944us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.609361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.609406 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:16:45.609515 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.609523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.609825 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.611213 (+ 1388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:45.611626 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.614274 (+ 2648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.618551 (+ 4277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.620938 (+ 2387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.623620 (+ 2682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.623887 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.623960 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.624135 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1650,"thread_start_us":2066,"threads_started":1}
I20260924 19:16:45.627837 27679 tablet_service.cc:1467] Processing CreateTablet for tablet b35f59d111c04d19b77135c0d84ba576 (DEFAULT_TABLE table=test-table [id=b587878f5ea7434a84e3f9cbed6e65af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:45.629503 27679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35f59d111c04d19b77135c0d84ba576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:45.629083 27810 tablet_service.cc:1467] Processing CreateTablet for tablet b35f59d111c04d19b77135c0d84ba576 (DEFAULT_TABLE table=test-table [id=b587878f5ea7434a84e3f9cbed6e65af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:45.630087 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.610090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.11.18.65:45805)
0924 19:16:45.623831 (+ 13741us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.623837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.623870 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:16:45.623951 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.623956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.624158 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.624372 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:45.624798 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.626352 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.627846 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.628979 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.629130 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.629374 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.629584 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.629753 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13563,"thread_start_us":13421,"threads_started":1}
I20260924 19:16:45.630312 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.609759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:40914)
0924 19:16:45.623019 (+ 13260us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:45.623039 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:45.623154 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:45.624365 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:45.624370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:45.624459 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:45.624645 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.624655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.626487 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.626495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.627653 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.627695 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.629146 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.629152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.629385 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.629431 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:16:45.629538 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:16:45.629874 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13137,"thread_start_us":13504,"threads_started":1}
I20260924 19:16:45.630578 27810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35f59d111c04d19b77135c0d84ba576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:45.647533 27547 tablet_service.cc:1467] Processing CreateTablet for tablet b35f59d111c04d19b77135c0d84ba576 (DEFAULT_TABLE table=test-table [id=b587878f5ea7434a84e3f9cbed6e65af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:16:45.649390 27547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35f59d111c04d19b77135c0d84ba576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:16:45.649734 27891 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Bootstrap starting.
I20260924 19:16:45.654013 27891 tablet_bootstrap.cc:654] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:45.655648 27891 log.cc:824] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:45.658895 27891 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: No bootstrap required, opened a new log
I20260924 19:16:45.659201 27891 ts_tablet_manager.cc:1397] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 19:16:45.660780 27893 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Bootstrap starting.
I20260924 19:16:45.664798 27893 tablet_bootstrap.cc:654] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:45.666445 27893 log.cc:824] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:45.670022 27893 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: No bootstrap required, opened a new log
I20260924 19:16:45.669798 27891 raft_consensus.cc:348] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.670251 27891 raft_consensus.cc:374] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:45.670301 27893 ts_tablet_manager.cc:1397] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 19:16:45.670352 27891 raft_consensus.cc:729] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9522b4c78c7c41f0b336a75bd653aa80, State: Initialized, Role: FOLLOWER
I20260924 19:16:45.671021 27891 consensus_queue.cc:260] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.672389 27875 heartbeater.cc:503] Master 127.11.18.126:41685 was elected leader, sending a full tablet report...
I20260924 19:16:45.672922 27891 ts_tablet_manager.cc:1428] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 19:16:45.676443 27895 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Bootstrap starting.
I20260924 19:16:45.680930 27895 tablet_bootstrap.cc:654] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Neither blocks nor log segments found. Creating new log.
I20260924 19:16:45.682359 27895 log.cc:824] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Log is configured to *not* fsync() on all Append() calls
I20260924 19:16:45.682602 27893 raft_consensus.cc:348] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.683037 27893 raft_consensus.cc:374] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:45.683168 27893 raft_consensus.cc:729] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee7f41ec93048299c414573d7704adc, State: Initialized, Role: FOLLOWER
I20260924 19:16:45.683872 27893 consensus_queue.cc:260] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.685060 27744 heartbeater.cc:503] Master 127.11.18.126:41685 was elected leader, sending a full tablet report...
I20260924 19:16:45.685760 27893 ts_tablet_manager.cc:1428] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260924 19:16:45.686023 27895 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: No bootstrap required, opened a new log
I20260924 19:16:45.686311 27895 ts_tablet_manager.cc:1397] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260924 19:16:45.694862 27895 raft_consensus.cc:348] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.695324 27895 raft_consensus.cc:374] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:16:45.695451 27895 raft_consensus.cc:729] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e44768b8934a4bb3d790647bf80838, State: Initialized, Role: FOLLOWER
I20260924 19:16:45.696079 27895 consensus_queue.cc:260] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.697932 27895 ts_tablet_manager.cc:1428] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260924 19:16:45.705391 27896 raft_consensus.cc:482] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:16:45.705673 27896 raft_consensus.cc:504] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.707376 27896 leader_election.cc:288] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319), 38e44768b8934a4bb3d790647bf80838 (127.11.18.65:45805)
I20260924 19:16:45.714984 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.707818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:36203 (local address 127.11.18.66:36319)
0924 19:16:45.708080 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.708088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.708186 (+ 98us) server_negotiation.cc:407] Connection header received
0924 19:16:45.708420 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.708426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.708498 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.708580 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:45.709261 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.710310 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.712623 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.714075 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.714206 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.714277 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.714627 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.714763 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:16:45.715060 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.707794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:36319 (local address 127.11.18.67:36203)
0924 19:16:45.708072 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:45.708093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:45.708263 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:45.708762 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:45.708771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:45.708855 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:45.709080 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.709093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.710439 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.710445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.712320 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.712339 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.714232 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.714240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.714436 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.714472 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:16:45.714576 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 19:16:45.714766 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 19:16:45.716166 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.708412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:52945 (local address 127.11.18.65:45805)
0924 19:16:45.708668 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.708678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.709494 (+ 816us) server_negotiation.cc:407] Connection header received
0924 19:16:45.709680 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.709686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.709753 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.709848 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:16:45.710341 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.711471 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.713943 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.715142 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.715421 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.715503 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.715871 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.715980 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:16:45.716312 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.708292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.11.18.67:52945)
0924 19:16:45.709375 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:45.709397 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:16:45.709540 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:16:45.709861 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:16:45.709866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:16:45.709890 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:16:45.710198 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.710209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.711624 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.711628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.713751 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:16:45.713768 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.715483 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.715488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:16:45.715661 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.715698 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:16:45.715810 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:16:45.715945 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":675,"threads_started":1}
I20260924 19:16:45.716102 27699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35f59d111c04d19b77135c0d84ba576" candidate_uuid: "9522b4c78c7c41f0b336a75bd653aa80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee7f41ec93048299c414573d7704adc" is_pre_election: true
I20260924 19:16:45.716722 27699 raft_consensus.cc:2454] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9522b4c78c7c41f0b336a75bd653aa80 in term 0.
I20260924 19:16:45.717299 27567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35f59d111c04d19b77135c0d84ba576" candidate_uuid: "9522b4c78c7c41f0b336a75bd653aa80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e44768b8934a4bb3d790647bf80838" is_pre_election: true
I20260924 19:16:45.717517 27764 leader_election.cc:302] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee7f41ec93048299c414573d7704adc, 9522b4c78c7c41f0b336a75bd653aa80; no voters:
I20260924 19:16:45.717944 27896 raft_consensus.cc:2781] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:16:45.717897 27567 raft_consensus.cc:2454] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9522b4c78c7c41f0b336a75bd653aa80 in term 0.
I20260924 19:16:45.718084 27896 raft_consensus.cc:482] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:16:45.718181 27896 raft_consensus.cc:3037] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:45.720517 27896 raft_consensus.cc:504] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.721227 27896 leader_election.cc:288] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [CANDIDATE]: Term 1 election: Requested vote from peers 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319), 38e44768b8934a4bb3d790647bf80838 (127.11.18.65:45805)
I20260924 19:16:45.722028 27699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35f59d111c04d19b77135c0d84ba576" candidate_uuid: "9522b4c78c7c41f0b336a75bd653aa80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee7f41ec93048299c414573d7704adc"
I20260924 19:16:45.722059 27567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35f59d111c04d19b77135c0d84ba576" candidate_uuid: "9522b4c78c7c41f0b336a75bd653aa80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e44768b8934a4bb3d790647bf80838"
I20260924 19:16:45.722270 27699 raft_consensus.cc:3037] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:45.722446 27567 raft_consensus.cc:3037] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:16:45.724709 27699 raft_consensus.cc:2454] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9522b4c78c7c41f0b336a75bd653aa80 in term 1.
I20260924 19:16:45.725171 27764 leader_election.cc:302] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee7f41ec93048299c414573d7704adc, 9522b4c78c7c41f0b336a75bd653aa80; no voters:
I20260924 19:16:45.725517 27896 raft_consensus.cc:2781] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:16:45.725852 27567 raft_consensus.cc:2454] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9522b4c78c7c41f0b336a75bd653aa80 in term 1.
I20260924 19:16:45.726111 27896 raft_consensus.cc:686] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 LEADER]: Becoming Leader. State: Replica: 9522b4c78c7c41f0b336a75bd653aa80, State: Running, Role: LEADER
I20260924 19:16:45.726758 27896 consensus_queue.cc:237] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:16:45.732050 27427 catalog_manager.cc:5495] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "9522b4c78c7c41f0b336a75bd653aa80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } health_report { overall_health: UNKNOWN } } }
W20260924 19:16:45.745410 27876 tablet.cc:2369] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:16:45.753207 27745 tablet.cc:2369] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:16:45.766584 27613 tablet.cc:2369] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:16:45.778366 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.772224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.11.18.67:46061)
0924 19:16:45.772487 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.772492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.772511 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:16:45.772605 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.772608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.772693 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.772837 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:16:45.773284 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.774184 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.775708 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.776128 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.776312 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.776921 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:16:45.777691 (+ 770us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:16:45.777823 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.777969 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.778046 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:16:45.788123 27567 raft_consensus.cc:1261] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Refusing update from remote peer 9522b4c78c7c41f0b336a75bd653aa80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:16:45.788130 27699 raft_consensus.cc:1261] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 1 FOLLOWER]: Refusing update from remote peer 9522b4c78c7c41f0b336a75bd653aa80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:16:45.789247 27896 consensus_queue.cc:1035] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:16:45.789608 27901 consensus_queue.cc:1035] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:16:45.818302 27426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:16:45.818704 27426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:16:45.823956 27426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Sending DeleteTablet for 3 replicas of tablet b35f59d111c04d19b77135c0d84ba576
I20260924 19:16:45.825547 27810 tablet_service.cc:1514] Processing DeleteTablet for tablet b35f59d111c04d19b77135c0d84ba576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:45 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:35632
I20260924 19:16:45.825621 27547 tablet_service.cc:1514] Processing DeleteTablet for tablet b35f59d111c04d19b77135c0d84ba576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:45 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:40914
I20260924 19:16:45.825680 27679 tablet_service.cc:1514] Processing DeleteTablet for tablet b35f59d111c04d19b77135c0d84ba576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:16:45 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:40520
I20260924 19:16:45.827024 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 27464
I20260924 19:16:45.842173 27913 tablet_replica.cc:330] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: stopping tablet replica
I20260924 19:16:45.842785 27913 raft_consensus.cc:2229] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:16:45.843627 27913 raft_consensus.cc:2258] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:16:45.846694 27913 ts_tablet_manager.cc:1905] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:16:45.856395 27913 ts_tablet_manager.cc:1918] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:16:45.856626 27913 log.cc:1196] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/b35f59d111c04d19b77135c0d84ba576
I20260924 19:16:45.857393 27913 ts_tablet_manager.cc:1939] T b35f59d111c04d19b77135c0d84ba576 P 9522b4c78c7c41f0b336a75bd653aa80: Deleting consensus metadata
I20260924 19:16:45.859609 27411 catalog_manager.cc:4841] TS 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061): tablet b35f59d111c04d19b77135c0d84ba576 (table test-table [id=b587878f5ea7434a84e3f9cbed6e65af]) successfully deleted
I20260924 19:16:45.860357 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:45.828952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.11.18.65:45805)
0924 19:16:45.829177 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:45.829184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:45.829203 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:16:45.829279 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:45.829287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:45.829429 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:45.829567 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:45.832606 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.834235 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.836880 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:45.837759 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:45.837917 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:45.848190 (+ 10273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:45.848212 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:45.848230 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:45.848281 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:45.853019 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.854240 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.854251 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.854256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.854461 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:45.855986 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:45.855995 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:45.856002 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:45.856843 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:45.857212 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:45.859187 (+ 1975us) server_negotiation.cc:299] Negotiation successful
0924 19:16:45.859496 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:16:45.861253 27607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:45.866102 27915 tablet_replica.cc:330] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: stopping tablet replica
I20260924 19:16:45.868296 27915 raft_consensus.cc:2229] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:16:45.869095 27915 raft_consensus.cc:2258] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:16:45.872465 27915 ts_tablet_manager.cc:1905] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:16:45.877961 27915 ts_tablet_manager.cc:1918] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:16:45.878116 27915 log.cc:1196] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/b35f59d111c04d19b77135c0d84ba576
I20260924 19:16:45.878545 27915 ts_tablet_manager.cc:1939] T b35f59d111c04d19b77135c0d84ba576 P 5ee7f41ec93048299c414573d7704adc: Deleting consensus metadata
I20260924 19:16:45.880322 27412 catalog_manager.cc:4841] TS 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319): tablet b35f59d111c04d19b77135c0d84ba576 (table test-table [id=b587878f5ea7434a84e3f9cbed6e65af]) successfully deleted
I20260924 19:16:46.011435 27914 tablet_replica.cc:330] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: stopping tablet replica
I20260924 19:16:46.011482 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 27464
W20260924 19:16:46.037273 27412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.18.65:45805 (error 108)
I20260924 19:16:46.038093 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 27616
I20260924 19:16:46.049932 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.049494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:54554)
0924 19:16:46.049676 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:46.049789 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 19:16:46.050297 27412 catalog_manager.cc:4584] TS 38e44768b8934a4bb3d790647bf80838 (127.11.18.65:45805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b35f59d111c04d19b77135c0d84ba576: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
I20260924 19:16:46.056875 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.039693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.11.18.66:36319)
0924 19:16:46.039874 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:46.039881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:46.039901 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:16:46.039965 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:46.039970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:46.040050 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:46.040148 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:46.042290 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:46.043524 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:46.045651 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:46.046338 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:46.046474 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:46.050497 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:46.050515 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:46.050534 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:46.050578 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:46.053954 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:46.054852 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:46.054859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:46.054861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:46.054999 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:46.055419 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:46.055422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:46.055424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:46.055818 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:46.056038 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:46.056442 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 19:16:46.056617 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:16:46.057790 27739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:46.104300 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.103763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:54558)
0924 19:16:46.104009 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:46.104123 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 19:16:46.105618 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.105100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:54562)
0924 19:16:46.105274 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:46.105436 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 19:16:46.157449 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 27616
I20260924 19:16:46.166401 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.165798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:54578)
0924 19:16:46.166060 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:46.166184 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 19:16:46.167791 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.167277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:54590)
0924 19:16:46.167471 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:46.167574 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 19:16:46.175467 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 27747
I20260924 19:16:46.194942 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.176603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.11.18.67:46061)
0924 19:16:46.176805 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:16:46.176809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:16:46.176881 (+ 72us) server_negotiation.cc:407] Connection header received
0924 19:16:46.177106 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:16:46.177110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:16:46.177185 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:16:46.177267 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:16:46.179481 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:46.180431 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:46.182772 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:16:46.183396 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:16:46.183524 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:16:46.187598 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:16:46.187617 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:16:46.187640 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:16:46.187698 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:16:46.191469 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:46.192470 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:46.192479 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:46.192481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:46.192648 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:16:46.193094 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:16:46.193097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:16:46.193099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:16:46.193570 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:16:46.193856 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 19:16:46.194271 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 19:16:46.194476 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:16:46.195887 27870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:46.249414 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.248821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:54592)
0924 19:16:46.249083 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:46.249205 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 19:16:46.251210 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.250699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:54604)
0924 19:16:46.250940 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:46.251031 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 19:16:46.330660 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 27747
I20260924 19:16:46.351264 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 27394
I20260924 19:16:46.352573 27457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:16:46.466845 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:16:46.466334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:45805 (local address 127.0.0.1:54620)
0924 19:16:46.466546 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:16:46.466659 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 19:16:46.466918 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 27394
I20260924 19:17:06.491042 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41685
--webserver_interface=127.11.18.126
--webserver_port=41341
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:41685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:06.841581 27921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:06.841885 27921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:06.842046 27921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:06.849620 27921 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:17:06.849727 27921 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:17:06.849788 27921 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:17:06.849874 27921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:06.849990 27921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:06.850113 27921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:06.850198 27921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:06.850281 27921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:06.860571 27921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:41685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=41341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:06.861946 27921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:06.863340 27921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:06.876545 27929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:06.876659 27927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:06.876732 27921 server_base.cc:1034] running on GCE node
W20260924 19:17:06.877064 27926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:06.878077 27921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:06.879782 27921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:06.881084 27921 hybrid_clock.cc:648] HybridClock initialized: now 1790277426881021 us; error 62 us; skew 500 ppm
I20260924 19:17:06.881531 27921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:06 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277426, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:06.892915 27921 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:17:06.894806 27921 webserver.cc:460] Webserver started at http://127.11.18.126:41341/ using document root <none> and password file <none>
I20260924 19:17:06.895658 27921 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:06.895757 27921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:06.902266 27921 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:17:06.905807 27936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:06.907420 27921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:17:06.907586 27921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "3926b46bbe3043378072f94aff49c6cb"
format_stamp: "Formatted at 2026-09-24 19:16:20 on dist-test-slave-0lgr"
server_key: "de62ab4948b81bd8268f162e3b86a57d"
server_key_iv: "cf488e459ff110f1c86d03671b870553"
server_key_version: "encryptionkey@0"
I20260924 19:17:06.908548 27921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:06.923313 27921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:06.928184 27921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:06.928438 27921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:06.949718 27921 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:41685
I20260924 19:17:06.949757 27987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:41685 every 8 connection(s)
I20260924 19:17:06.951403 27921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:06.956816 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 27921
I20260924 19:17:06.958046 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:45805
--local_ip_for_outbound_sockets=127.11.18.65
--tserver_master_addrs=127.11.18.126:41685
--webserver_port=36283
--webserver_interface=127.11.18.65
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:17:06.960513 27988 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:17:06.966635 27988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Bootstrap starting.
I20260924 19:17:06.990410 27988 log.cc:824] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:07.008826 27988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:17:07.009539 27988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Bootstrap complete.
I20260924 19:17:07.020448 27988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:17:07.021524 27988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3926b46bbe3043378072f94aff49c6cb, State: Initialized, Role: FOLLOWER
I20260924 19:17:07.022302 27988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:17:07.022567 27988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:07.022711 27988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:07.022907 27988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:17:07.027420 27988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:17:07.027987 27988 leader_election.cc:302] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3926b46bbe3043378072f94aff49c6cb; no voters:
I20260924 19:17:07.028685 27988 leader_election.cc:288] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:17:07.028869 27992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:17:07.030321 27992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [term 3 LEADER]: Becoming Leader. State: Replica: 3926b46bbe3043378072f94aff49c6cb, State: Running, Role: LEADER
I20260924 19:17:07.030898 27992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } }
I20260924 19:17:07.031172 27988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:07.033421 27994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3926b46bbe3043378072f94aff49c6cb. Latest consensus state: current_term: 3 leader_uuid: "3926b46bbe3043378072f94aff49c6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } } }
I20260924 19:17:07.033759 27994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:07.034350 27993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3926b46bbe3043378072f94aff49c6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3926b46bbe3043378072f94aff49c6cb" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41685 } } }
I20260924 19:17:07.034685 27993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:07.040102 27997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:07.052464 27997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:07.055016 27997 catalog_manager.cc:1259] Loaded cluster ID: 96a21145d0a94672aea21098a6bb242e
I20260924 19:17:07.055240 27997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:07.061934 27997 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:07.062510 28010 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:17:07.064103 27997 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Loaded TSK: 0
I20260924 19:17:07.064288 27997 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3926b46bbe3043378072f94aff49c6cb: Loaded TSK: 1
F20260924 19:17:07.065472 27997 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fda8a8259ad google::LogMessage::Fail() at ??:0
@ 0x7fda8a82888a google::LogMessage::SendToLog() at ??:0
@ 0x7fda8a8254de google::LogMessage::Flush() at ??:0
@ 0x7fda8a826a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fda8e5897d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fda8e57873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdaa02541c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdaa0253783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdaa02df2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fdaa02df0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fdaa03295e9 std::function<>::operator()() at ??:0
@ 0x7fdaa0257cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fdaa0255c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdaa02dd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fdaa02dd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fda8d471345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fda8d47491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fda8d474799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fda8d44feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fda907296db start_thread at ??:0
@ 0x7fda896bf61f clone at ??:0
W20260924 19:17:07.404881 27990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:07.405195 27990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:07.405373 27990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:07.412984 27990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:07.413123 27990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:07.413337 27990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:17:07.423599 27990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:45805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=36283
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:07.424968 27990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:07.426363 27990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:07.440936 28036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:07.440980 28039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:07.441023 28037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:07.442054 27990 server_base.cc:1034] running on GCE node
I20260924 19:17:07.442816 27990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:07.443892 27990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:07.445168 27990 hybrid_clock.cc:648] HybridClock initialized: now 1790277427445106 us; error 74 us; skew 500 ppm
I20260924 19:17:07.445637 27990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:07 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277427, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:07.449683 27990 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:17:07.451442 27990 webserver.cc:460] Webserver started at http://127.11.18.65:36283/ using document root <none> and password file <none>
I20260924 19:17:07.452199 27990 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:07.452368 27990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:07.458390 27990 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:17:07.461846 28046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:07.463630 27990 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:17:07.463838 27990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "38e44768b8934a4bb3d790647bf80838"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "cb99b8fe5721cd6c727d7cbbfce0d9c7"
server_key_iv: "4db058db33d20b0428c4ca7eac68494f"
server_key_version: "encryptionkey@0"
I20260924 19:17:07.464968 27990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:07.488204 27990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:07.493152 27990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:07.493510 27990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:07.494647 27990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:07.496723 28053 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 19:17:07.502861 27990 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:17:07.503067 27990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260924 19:17:07.503242 27990 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:17:07.507000 27990 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:17:07.507445 27990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.003s
I20260924 19:17:07.507829 28053 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Bootstrap starting.
I20260924 19:17:07.539053 28053 log.cc:824] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:07.554353 28053 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:17:07.555019 28053 tablet_bootstrap.cc:492] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Bootstrap complete.
I20260924 19:17:07.556582 27990 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:45805
I20260924 19:17:07.556622 28160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:45805 every 8 connection(s)
I20260924 19:17:07.556753 28053 ts_tablet_manager.cc:1397] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Time spent bootstrapping tablet: real 0.049s user 0.045s sys 0.000s
I20260924 19:17:07.558326 27990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:17:07.560357 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 27990
I20260924 19:17:07.561225 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:36319
--local_ip_for_outbound_sockets=127.11.18.66
--tserver_master_addrs=127.11.18.126:41685
--webserver_port=37485
--webserver_interface=127.11.18.66
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:17:07.564044 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:07.561686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:57087)
0924 19:17:07.562642 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:07.562995 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":443,"threads_started":1}
W20260924 19:17:07.564792 28161 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:41685 (0 consecutive failures): Network error: Failed to ping master at 127.11.18.126:41685: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
I20260924 19:17:07.565908 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:07.565418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:37003)
0924 19:17:07.565684 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:07.565775 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 19:17:07.566870 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:07.566505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:58449)
0924 19:17:07.566656 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:07.566748 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:17:07.567091 28161 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:17:07.569378 28053 raft_consensus.cc:348] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:17:07.570950 28053 raft_consensus.cc:729] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e44768b8934a4bb3d790647bf80838, State: Initialized, Role: FOLLOWER
I20260924 19:17:07.571672 28053 consensus_queue.cc:260] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:17:07.573531 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:07.573078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:42409)
0924 19:17:07.573256 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:07.573390 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 19:17:07.573655 28053 ts_tablet_manager.cc:1428] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838: Time spent starting tablet: real 0.017s user 0.016s sys 0.004s
I20260924 19:17:07.574731 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:07.574251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:39685)
0924 19:17:07.574449 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:07.574553 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 19:17:07.919009 28164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:07.919432 28164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:07.919615 28164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:07.930083 28164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:07.930204 28164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:07.930408 28164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:17:07.940868 28164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:36319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=37485
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:07.942338 28164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:07.943712 28164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:07.957803 28170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:07.957856 28173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:07.957866 28171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:07.958792 28164 server_base.cc:1034] running on GCE node
I20260924 19:17:07.959440 28164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:07.960429 28164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:07.961658 28164 hybrid_clock.cc:648] HybridClock initialized: now 1790277427961610 us; error 47 us; skew 500 ppm
I20260924 19:17:07.961992 28164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:07 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277427, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:07.965451 28164 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:17:07.967181 28164 webserver.cc:460] Webserver started at http://127.11.18.66:37485/ using document root <none> and password file <none>
I20260924 19:17:07.967918 28164 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:07.968075 28164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:07.973786 28164 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:17:07.976930 28180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:07.978590 28164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:07.978809 28164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "5ee7f41ec93048299c414573d7704adc"
format_stamp: "Formatted at 2026-09-24 19:16:21 on dist-test-slave-0lgr"
server_key: "453dea7a3bb629d55f7665e292bc1960"
server_key_iv: "9b5936b41cf0bc3dabdacc42d8a5476c"
server_key_version: "encryptionkey@0"
I20260924 19:17:07.979935 28164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:08.008059 28164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:08.012734 28164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:08.013041 28164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:08.014117 28164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:08.015545 28164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:08.015679 28164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:08.015817 28164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:08.015905 28164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:08.055801 28164 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:36319
I20260924 19:17:08.055900 28292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:36319 every 8 connection(s)
I20260924 19:17:08.057520 28164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:17:08.063129 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.061035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:56465)
0924 19:17:08.061883 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.062131 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":371,"threads_started":1}
I20260924 19:17:08.063931 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 28164
W20260924 19:17:08.064097 28293 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:41685 (0 consecutive failures): Network error: Failed to ping master at 127.11.18.126:41685: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
I20260924 19:17:08.064808 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:46061
--local_ip_for_outbound_sockets=127.11.18.67
--tserver_master_addrs=127.11.18.126:41685
--webserver_port=37599
--webserver_interface=127.11.18.67
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:17:08.065284 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.064838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:35513)
0924 19:17:08.065042 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.065159 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 19:17:08.066447 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.066033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:34361)
0924 19:17:08.066226 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.066327 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260924 19:17:08.066735 28293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:17:08.414819 28296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:08.415115 28296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:08.415215 28296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:08.423046 28296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:08.423185 28296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:08.423305 28296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:17:08.433683 28296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:46061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=37599
--tserver_master_addrs=127.11.18.126:41685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:08.434980 28296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:08.436272 28296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:08.450690 28301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:08.451030 28304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:08.450814 28302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:08.451689 28296 server_base.cc:1034] running on GCE node
I20260924 19:17:08.452382 28296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:08.453696 28296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:08.454969 28296 hybrid_clock.cc:648] HybridClock initialized: now 1790277428454900 us; error 53 us; skew 500 ppm
I20260924 19:17:08.455348 28296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:08 dist-test-slave-0lgr krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277428, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:08.459023 28296 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:17:08.460844 28296 webserver.cc:460] Webserver started at http://127.11.18.67:37599/ using document root <none> and password file <none>
I20260924 19:17:08.461616 28296 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:08.461735 28296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:08.467257 28296 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:17:08.470098 28311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:08.471652 28296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:08.471868 28296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "9522b4c78c7c41f0b336a75bd653aa80"
format_stamp: "Formatted at 2026-09-24 19:16:22 on dist-test-slave-0lgr"
server_key: "9ca3a2b0bbcb8f95b274d3cb47e501a8"
server_key_iv: "86c1ed8396d6f6aef25529deab69329a"
server_key_version: "encryptionkey@0"
I20260924 19:17:08.472949 28296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:08.511054 28296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:08.515769 28296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:08.516106 28296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:08.517228 28296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:08.518730 28296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:08.518865 28296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:08.519011 28296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:08.519106 28296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:08.558096 28296 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:46061
I20260924 19:17:08.558118 28423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:46061 every 8 connection(s)
I20260924 19:17:08.560029 28296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:17:08.562335 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 28296
I20260924 19:17:08.565974 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.563405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:51983)
0924 19:17:08.564413 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.564709 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":404,"threads_started":1}
W20260924 19:17:08.566701 28424 heartbeater.cc:644] Failed to heartbeat to 127.11.18.126:41685 (0 consecutive failures): Network error: Failed to ping master at 127.11.18.126:41685: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
I20260924 19:17:08.567634 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.567228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:34747)
0924 19:17:08.567410 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.567497 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 19:17:08.568882 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.568398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:34151)
0924 19:17:08.568595 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.568748 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 19:17:08.569182 28424 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:17:08.576507 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.575641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:39999)
0924 19:17:08.576157 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.576292 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
I20260924 19:17:08.938283 28429 raft_consensus.cc:482] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:08.938716 28429 raft_consensus.cc:504] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:17:08.940912 28429 leader_election.cc:288] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061), 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319)
Sep 24 19:17:08 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277427, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
Sep 24 19:17:08 dist-test-slave-0lgr krb5kdc[26840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277427, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:17:08.969913 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.941591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:37569 (local address 127.11.18.67:46061)
0924 19:17:08.943277 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:08.943293 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:08.943342 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:17:08.946023 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:08.946038 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:08.946643 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:08.947098 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:08.950345 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:08.952474 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:08.955504 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:08.956600 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:08.956818 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:08.962466 (+ 5648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:08.962475 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:08.962500 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:08.962544 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:08.965820 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:08.966717 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:08.966725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:08.966730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:08.966903 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:08.967364 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:08.967368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:08.967370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:08.967929 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:08.968150 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:08.969013 (+ 863us) server_negotiation.cc:299] Negotiation successful
0924 19:17:08.969451 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1455,"thread_start_us":1230,"threads_started":1}
I20260924 19:17:08.970121 28432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.941603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:46263 (local address 127.11.18.66:36319)
0924 19:17:08.942487 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:08.942515 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:08.943596 (+ 1081us) server_negotiation.cc:407] Connection header received
0924 19:17:08.946038 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:08.946048 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:08.946642 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:08.947098 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:08.950346 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:08.952474 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:08.955504 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:08.956604 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:08.956817 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:08.962868 (+ 6051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:08.962876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:08.962889 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:08.962930 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:08.966202 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:08.966978 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:08.966984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:08.966989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:08.967168 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:08.967617 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:08.967621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:08.967623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:08.968129 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:08.968345 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:08.969054 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 19:17:08.969671 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":386,"threads_started":1}
I20260924 19:17:08.970323 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.941165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46061 (local address 127.11.18.65:37569)
0924 19:17:08.942436 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.942496 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:08.943160 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:08.947335 (+ 4175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:08.947348 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:08.948762 (+ 1414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:08.950066 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:08.950099 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:08.952743 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:08.952748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:08.955302 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:08.955320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:08.957068 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:08.957076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:08.957499 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:08.958539 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:08.958587 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:08.962087 (+ 3500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:08.966001 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:08.966011 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:08.966027 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:08.966542 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:08.967026 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:08.967031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:08.967034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:08.967233 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:08.968074 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:08.968082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:08.968388 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 19:17:08.968772 (+ 384us) client_negotiation.cc:240] Negotiation successful
0924 19:17:08.969871 (+ 1099us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":417,"spinlock_wait_cycles":54528}
I20260924 19:17:08.970932 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:08.941395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:36319 (local address 127.11.18.65:46263)
0924 19:17:08.943465 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:08.943483 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:08.943621 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:08.947335 (+ 3714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:08.947348 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:08.948762 (+ 1414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:08.950066 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:08.950099 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:08.952742 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:08.952748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:08.955302 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:08.955320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:08.957068 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:08.957076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:08.957502 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:08.958541 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:08.958587 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:08.962571 (+ 3984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:08.966372 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:08.966379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:08.966383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:08.966806 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:08.967298 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:08.967305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:08.967309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:08.967468 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:08.968270 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:08.968275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:08.968483 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 19:17:08.968771 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 19:17:08.970613 (+ 1842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1924,"mutex_wait_us":721,"spinlock_wait_cycles":25728,"thread_start_us":1757,"threads_started":1}
W20260924 19:17:08.973963 28047 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
W20260924 19:17:08.974313 28048 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
I20260924 19:17:08.974458 28048 leader_election.cc:302] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38e44768b8934a4bb3d790647bf80838; no voters: 5ee7f41ec93048299c414573d7704adc, 9522b4c78c7c41f0b336a75bd653aa80
I20260924 19:17:08.974826 28429 raft_consensus.cc:2726] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:17:09.068485 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:09.067370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:40361)
0924 19:17:09.067893 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:09.068116 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
I20260924 19:17:09.570796 28434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:09.569899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:59491)
0924 19:17:09.570435 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:09.570558 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":268,"threads_started":1}
I20260924 19:17:09.578358 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:09.577574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:60587)
0924 19:17:09.578082 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:09.578191 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":246,"threads_started":1}
I20260924 19:17:10.070797 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:10.069901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:45947)
0924 19:17:10.070437 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:10.070580 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":286,"threads_started":1}
I20260924 19:17:10.550645 28437 raft_consensus.cc:482] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:10.550839 28437 raft_consensus.cc:504] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:17:10.551509 28437 leader_election.cc:288] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061), 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319)
W20260924 19:17:10.552606 28048 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
W20260924 19:17:10.552809 28047 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
I20260924 19:17:10.552954 28047 leader_election.cc:302] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38e44768b8934a4bb3d790647bf80838; no voters: 5ee7f41ec93048299c414573d7704adc, 9522b4c78c7c41f0b336a75bd653aa80
I20260924 19:17:10.553284 28437 raft_consensus.cc:2726] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:17:10.572764 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:10.571952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:42189)
0924 19:17:10.572462 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:10.572574 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":255,"threads_started":1}
I20260924 19:17:10.580276 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:10.579397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:36537)
0924 19:17:10.579961 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:10.580073 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":283,"threads_started":1}
I20260924 19:17:11.072885 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:11.071917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:60063)
0924 19:17:11.072571 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:11.072690 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":370,"threads_started":1}
I20260924 19:17:11.574553 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:11.573838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:56925)
0924 19:17:11.574297 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:11.574393 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260924 19:17:11.582212 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:11.581382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:48603)
0924 19:17:11.581911 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:11.582020 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":317,"threads_started":1}
I20260924 19:17:12.074920 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:12.074009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:42159)
0924 19:17:12.074564 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:12.074658 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260924 19:17:12.576467 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:12.575616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:35921)
0924 19:17:12.576189 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:12.576281 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":309,"threads_started":1}
I20260924 19:17:12.584298 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:12.583431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:48475)
0924 19:17:12.583981 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:12.584095 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":322,"threads_started":1}
I20260924 19:17:13.077004 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:13.076059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:56419)
0924 19:17:13.076639 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:13.076819 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":261,"threads_started":1}
I20260924 19:17:13.314667 28447 raft_consensus.cc:482] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:13.314966 28447 raft_consensus.cc:504] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:17:13.315816 28447 leader_election.cc:288] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061), 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319)
W20260924 19:17:13.316838 28047 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
W20260924 19:17:13.317056 28048 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
I20260924 19:17:13.317262 28048 leader_election.cc:302] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38e44768b8934a4bb3d790647bf80838; no voters: 5ee7f41ec93048299c414573d7704adc, 9522b4c78c7c41f0b336a75bd653aa80
I20260924 19:17:13.317626 28447 raft_consensus.cc:2726] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:17:13.578486 28448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:13.577617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:47021)
0924 19:17:13.578205 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:13.578327 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":319,"threads_started":1}
I20260924 19:17:13.586364 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:13.585500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:50569)
0924 19:17:13.586039 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:13.586151 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":247,"threads_started":1}
I20260924 19:17:14.078986 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:14.078027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:52331)
0924 19:17:14.078551 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:14.078639 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":306,"threads_started":1}
I20260924 19:17:14.580399 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:14.579587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:56737)
0924 19:17:14.580132 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:14.580218 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":294,"threads_started":1}
I20260924 19:17:14.588236 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:14.587431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:38335)
0924 19:17:14.587949 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:14.588035 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":302,"threads_started":1}
I20260924 19:17:15.080868 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:15.080038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:46843)
0924 19:17:15.080569 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:15.080667 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":232,"threads_started":1}
I20260924 19:17:15.582456 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:15.581603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:48693)
0924 19:17:15.582147 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:15.582237 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":255,"threads_started":1}
I20260924 19:17:15.590186 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:15.589255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:45151)
0924 19:17:15.589900 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:15.589995 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":372,"threads_started":1}
I20260924 19:17:16.083034 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:16.082059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:57543)
0924 19:17:16.082643 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:16.082779 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":307,"threads_started":1}
I20260924 19:17:16.584525 28457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:16.583637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:56099)
0924 19:17:16.584240 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:16.584342 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":364,"threads_started":1}
I20260924 19:17:16.592170 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:16.591174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:43357)
0924 19:17:16.591752 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:16.591961 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":345,"threads_started":1}
I20260924 19:17:17.085397 28459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:17.084215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:51783)
0924 19:17:17.084932 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:17.085135 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":267,"threads_started":1}
I20260924 19:17:17.140134 28460 raft_consensus.cc:482] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:17.140329 28460 raft_consensus.cc:504] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:17:17.141120 28460 leader_election.cc:288] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061), 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319)
W20260924 19:17:17.142226 28047 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
W20260924 19:17:17.142413 28048 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
I20260924 19:17:17.142530 28048 leader_election.cc:302] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38e44768b8934a4bb3d790647bf80838; no voters: 5ee7f41ec93048299c414573d7704adc, 9522b4c78c7c41f0b336a75bd653aa80
I20260924 19:17:17.142768 28460 raft_consensus.cc:2726] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:17:17.586699 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:17.585652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:52393)
0924 19:17:17.586319 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:17.586420 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":332,"threads_started":1}
I20260924 19:17:17.594414 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:17.593481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:37791)
0924 19:17:17.594041 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:17.594192 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":296,"threads_started":1}
I20260924 19:17:18.087172 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:18.086418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:50415)
0924 19:17:18.086837 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:18.086924 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":225,"threads_started":1}
I20260924 19:17:18.588691 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:18.587891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:56849)
0924 19:17:18.588371 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:18.588462 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":266,"threads_started":1}
I20260924 19:17:18.596261 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:18.595456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:34555)
0924 19:17:18.595946 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:18.596040 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":270,"threads_started":1}
I20260924 19:17:19.089313 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:19.088278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:36123)
0924 19:17:19.088885 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:19.089049 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":272,"threads_started":1}
I20260924 19:17:19.590521 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:19.589720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:34833)
0924 19:17:19.590232 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:19.590347 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":242,"threads_started":1}
I20260924 19:17:19.598150 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:19.597412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:35917)
0924 19:17:19.597904 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:19.597987 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":263,"threads_started":1}
I20260924 19:17:20.091281 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:20.090383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:47491)
0924 19:17:20.090957 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:20.091079 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":299,"threads_started":1}
I20260924 19:17:20.592717 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:20.591803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:39783)
0924 19:17:20.592402 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:20.592497 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":323,"threads_started":1}
I20260924 19:17:20.600220 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:20.599212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:42691)
0924 19:17:20.599913 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:20.600001 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":455,"threads_started":1}
I20260924 19:17:21.093204 28472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:21.092303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:45593)
0924 19:17:21.092848 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:21.092952 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260924 19:17:21.594952 28473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:21.593978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:55453)
0924 19:17:21.594660 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:21.594777 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":448,"threads_started":1}
I20260924 19:17:21.602391 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:21.601435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:34025)
0924 19:17:21.602083 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:21.602184 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":422,"threads_started":1}
I20260924 19:17:22.095173 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:22.094377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:47557)
0924 19:17:22.094877 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:22.094970 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":246,"threads_started":1}
I20260924 19:17:22.596971 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:22.596040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:42213)
0924 19:17:22.596660 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:22.596769 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":320,"threads_started":1}
I20260924 19:17:22.604729 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:22.603798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.65:42839)
0924 19:17:22.604426 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:22.604523 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":306,"threads_started":1}
I20260924 19:17:23.097411 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:23.096352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.66:33523)
0924 19:17:23.097013 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:23.097129 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":361,"threads_started":1}
I20260924 19:17:23.105914 28479 raft_consensus.cc:482] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:23.106662 28479 raft_consensus.cc:504] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9522b4c78c7c41f0b336a75bd653aa80" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46061 } } peers { permanent_uuid: "5ee7f41ec93048299c414573d7704adc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36319 } } peers { permanent_uuid: "38e44768b8934a4bb3d790647bf80838" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 45805 } }
I20260924 19:17:23.107546 28479 leader_election.cc:288] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061), 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319)
W20260924 19:17:23.108525 28047 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9522b4c78c7c41f0b336a75bd653aa80 (127.11.18.67:46061): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
W20260924 19:17:23.108872 28048 leader_election.cc:341] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee7f41ec93048299c414573d7704adc (127.11.18.66:36319): Not found: Tablet not found: b35f59d111c04d19b77135c0d84ba576
I20260924 19:17:23.109025 28048 leader_election.cc:302] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38e44768b8934a4bb3d790647bf80838; no voters: 5ee7f41ec93048299c414573d7704adc, 9522b4c78c7c41f0b336a75bd653aa80
I20260924 19:17:23.109313 28479 raft_consensus.cc:2726] T b35f59d111c04d19b77135c0d84ba576 P 38e44768b8934a4bb3d790647bf80838 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:17:23.571622 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 27990
I20260924 19:17:23.596418 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:23.572962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.11.18.65:45805)
0924 19:17:23.574025 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:23.574038 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:23.574079 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:17:23.574249 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:23.574265 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:23.574617 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:23.574833 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:23.576351 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:23.577614 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:23.579099 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:23.579713 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:23.579849 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:23.589282 (+ 9433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:23.589302 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:23.589386 (+ 84us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:23.589452 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:23.593029 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:23.594034 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:23.594043 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:23.594045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:23.594204 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:23.594632 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:23.594640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:23.594648 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:23.595065 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:23.595442 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:23.595778 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 19:17:23.596065 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":615,"threads_started":1}
I20260924 19:17:23.598040 28156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:23.599422 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:23.598459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41685 (local address 127.11.18.67:60541)
0924 19:17:23.599072 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:23.599189 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.126:41685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":290,"threads_started":1}
I20260924 19:17:23.693605 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 27990
I20260924 19:17:23.712819 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 28164
I20260924 19:17:23.727516 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:23.713669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.11.18.66:36319)
0924 19:17:23.714216 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:23.714220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:23.714237 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:17:23.714295 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:23.714298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:23.714360 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:23.714454 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:23.715928 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:23.716730 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:23.718178 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:23.718626 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:23.718728 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:23.721502 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:23.721507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:23.721510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:23.721552 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:23.724639 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:23.725401 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:23.725404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:23.725407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:23.725548 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:23.726009 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:23.726012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:23.726014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:23.726287 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:23.726489 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:23.726952 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 19:17:23.727125 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":330,"threads_started":1}
I20260924 19:17:23.728247 28288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:23.816635 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 28164
I20260924 19:17:23.836658 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 28296
I20260924 19:17:23.852387 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:23.837754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.11.18.67:46061)
0924 19:17:23.838616 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:23.838621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:23.838641 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:17:23.838703 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:23.838707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:23.838774 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:23.838967 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:23.840940 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:23.841902 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:23.843166 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:23.843607 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:23.843719 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:23.846530 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:23.846535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:23.846538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:23.846574 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:23.849637 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:23.850326 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:23.850331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:23.850333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:23.850482 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:23.850843 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:23.850847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:23.850850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:23.851168 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:23.851412 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:23.851716 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:17:23.851945 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":447,"threads_started":1}
I20260924 19:17:23.853155 28419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:23.942057 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 28296
2026-09-24T19:17:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65260 ms)
[----------] 30 tests from SecurityITest (315084 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:24 dist-test-slave-0lgr krb5kdc[28490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:24 dist-test-slave-0lgr krb5kdc[28490](info): set up 2 sockets
Sep 24 19:17:24 dist-test-slave-0lgr krb5kdc[28490](info): commencing operation
krb5kdc: starting...
W20260924 19:17:25.189831 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:25 dist-test-slave-0lgr krb5kdc[28490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:17:25.450304 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42201
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:25.796356 28503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:25.796674 28503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:25.796877 28503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:25.804463 28503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:25.804605 28503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:25.804705 28503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:25.804795 28503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:25.804859 28503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:25.815647 28503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:42201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:25.817111 28503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:25.818485 28503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:25.830973 28509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:25.831113 28508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:25.831089 28511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:25.832521 28503 server_base.cc:1034] running on GCE node
I20260924 19:17:25.833438 28503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:25.835023 28503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:25.836267 28503 hybrid_clock.cc:648] HybridClock initialized: now 1790277445836226 us; error 36 us; skew 500 ppm
I20260924 19:17:25.836615 28503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:25 dist-test-slave-0lgr krb5kdc[28490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:25.839956 28503 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:17:25.841572 28503 webserver.cc:460] Webserver started at http://127.11.18.126:43567/ using document root <none> and password file <none>
I20260924 19:17:25.842278 28503 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:25.842427 28503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:25.842733 28503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:25.845943 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "6d3df507f3824b3cbc39df3bfc1b204d"
format_stamp: "Formatted at 2026-09-24 19:17:25 on dist-test-slave-0lgr"
server_key: "09841ae76ada8bc0b95ffea9762225b1"
server_key_iv: "f6bb08c923910259094ed329db29d80e"
server_key_version: "encryptionkey@0"
I20260924 19:17:25.846704 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "6d3df507f3824b3cbc39df3bfc1b204d"
format_stamp: "Formatted at 2026-09-24 19:17:25 on dist-test-slave-0lgr"
server_key: "09841ae76ada8bc0b95ffea9762225b1"
server_key_iv: "f6bb08c923910259094ed329db29d80e"
server_key_version: "encryptionkey@0"
I20260924 19:17:25.851948 28503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:17:25.855646 28518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:25.857237 28503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:17:25.857470 28503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "6d3df507f3824b3cbc39df3bfc1b204d"
format_stamp: "Formatted at 2026-09-24 19:17:25 on dist-test-slave-0lgr"
server_key: "09841ae76ada8bc0b95ffea9762225b1"
server_key_iv: "f6bb08c923910259094ed329db29d80e"
server_key_version: "encryptionkey@0"
I20260924 19:17:25.857689 28503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:25.888351 28503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:25.892763 28503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:25.893014 28503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:25.913779 28503 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:42201
I20260924 19:17:25.913833 28569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:42201 every 8 connection(s)
I20260924 19:17:25.915278 28503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:25.916950 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 28503
I20260924 19:17:25.917179 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:25.917918 11337 external_mini_cluster.cc:1428] Setting key 23ae30cd40f0a1ea9375d4835c080f9b
I20260924 19:17:25.921715 28570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:17:25 dist-test-slave-0lgr krb5kdc[28490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:25.933717 28570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d: Bootstrap starting.
I20260924 19:17:25.937883 28570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:25.939221 28570 log.cc:824] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:25.941214 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:25.919919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.11.18.126:42201)
0924 19:17:25.920745 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:25.920758 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:25.920799 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:17:25.922719 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:25.922742 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:25.923134 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:25.923537 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:25.925142 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:25.926592 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:25.927938 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:25.928800 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:25.929087 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:25.933379 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:25.933394 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:25.933416 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:25.933463 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:25.936699 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:25.937516 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:25.937524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:25.937530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:25.937696 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:25.938286 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:25.938292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:25.938295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:25.938842 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:25.939150 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:25.939655 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 19:17:25.940146 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":348,"threads_started":1}
I20260924 19:17:25.942729 28570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d: No bootstrap required, opened a new log
I20260924 19:17:25.951021 28570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3df507f3824b3cbc39df3bfc1b204d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42201 } }
I20260924 19:17:25.951493 28570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:25.951601 28570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3df507f3824b3cbc39df3bfc1b204d, State: Initialized, Role: FOLLOWER
I20260924 19:17:25.952302 28570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3df507f3824b3cbc39df3bfc1b204d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42201 } }
I20260924 19:17:25.952549 28570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:25.952687 28570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:25.952873 28570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:25.955209 28570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3df507f3824b3cbc39df3bfc1b204d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42201 } }
I20260924 19:17:25.955740 28570 leader_election.cc:302] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d3df507f3824b3cbc39df3bfc1b204d; no voters:
I20260924 19:17:25.956480 28570 leader_election.cc:288] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:25.956725 28575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:25.957471 28575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [term 1 LEADER]: Becoming Leader. State: Replica: 6d3df507f3824b3cbc39df3bfc1b204d, State: Running, Role: LEADER
I20260924 19:17:25.958158 28575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3df507f3824b3cbc39df3bfc1b204d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42201 } }
I20260924 19:17:25.959048 28570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:25.961236 28577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d3df507f3824b3cbc39df3bfc1b204d. Latest consensus state: current_term: 1 leader_uuid: "6d3df507f3824b3cbc39df3bfc1b204d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3df507f3824b3cbc39df3bfc1b204d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42201 } } }
I20260924 19:17:25.961724 28577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:25.963117 28576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d3df507f3824b3cbc39df3bfc1b204d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3df507f3824b3cbc39df3bfc1b204d" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42201 } } }
I20260924 19:17:25.963536 28576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:25.964826 28583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:25.969676 28583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:25.979844 28583 catalog_manager.cc:1347] Generated new cluster ID: f12ce426fa044193a680d632a733b95c
I20260924 19:17:25.980072 28583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:26.010180 28583 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:26.011781 28583 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:26.023442 28583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d: Generated new TSK 0
I20260924 19:17:26.024504 28583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:17:26.101789 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42201
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:26.462023 28596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:26.462347 28596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:26.462512 28596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:26.469995 28596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:26.470161 28596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:26.470343 28596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:17:26.480335 28596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:26.481729 28596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:26.483079 28596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:26.497483 28601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:26.497835 28604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:26.497974 28602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:26.498927 28596 server_base.cc:1034] running on GCE node
I20260924 19:17:26.499953 28596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:26.501138 28596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:26.502444 28596 hybrid_clock.cc:648] HybridClock initialized: now 1790277446502395 us; error 93 us; skew 500 ppm
I20260924 19:17:26.502817 28596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:26 dist-test-slave-0lgr krb5kdc[28490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:26.506408 28596 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:17:26.508047 28596 webserver.cc:460] Webserver started at http://127.11.18.65:41069/ using document root <none> and password file <none>
I20260924 19:17:26.508747 28596 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:26.508862 28596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:26.509120 28596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:26.512212 28596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "cb214bd1463c479eba5433b88144000d"
format_stamp: "Formatted at 2026-09-24 19:17:26 on dist-test-slave-0lgr"
server_key: "cd32a79b82031cb07519e21467a7909d"
server_key_iv: "9109b3d2f072d454987449967b163cee"
server_key_version: "encryptionkey@0"
I20260924 19:17:26.513007 28596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "cb214bd1463c479eba5433b88144000d"
format_stamp: "Formatted at 2026-09-24 19:17:26 on dist-test-slave-0lgr"
server_key: "cd32a79b82031cb07519e21467a7909d"
server_key_iv: "9109b3d2f072d454987449967b163cee"
server_key_version: "encryptionkey@0"
I20260924 19:17:26.518720 28596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:17:26.522696 28611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:26.524943 28596 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:17:26.525159 28596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "cb214bd1463c479eba5433b88144000d"
format_stamp: "Formatted at 2026-09-24 19:17:26 on dist-test-slave-0lgr"
server_key: "cd32a79b82031cb07519e21467a7909d"
server_key_iv: "9109b3d2f072d454987449967b163cee"
server_key_version: "encryptionkey@0"
I20260924 19:17:26.525496 28596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:26.546439 28596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:26.551481 28596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:26.551852 28596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:26.553031 28596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:26.554631 28596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:26.554713 28596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:26.554858 28596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:26.554925 28596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:26.593501 28596 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:38369
I20260924 19:17:26.593636 28723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:38369 every 8 connection(s)
I20260924 19:17:26.594858 28596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:17:26.603195 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 28596
I20260924 19:17:26.603463 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:17:26.604218 11337 external_mini_cluster.cc:1428] Setting key e7188db1a829369a5f33c83e4d8dbab7
Sep 24 19:17:26 dist-test-slave-0lgr krb5kdc[28490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:26.617004 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:26.597667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:44441 (local address 127.11.18.126:42201)
0924 19:17:26.598187 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:26.598192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:26.599081 (+ 889us) server_negotiation.cc:407] Connection header received
0924 19:17:26.600992 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:26.600998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:26.601098 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:26.601216 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:26.602813 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:26.603641 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:26.605469 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:26.605900 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:26.606049 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:26.610188 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:26.610195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:26.610198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:26.610232 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:26.613398 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:26.614327 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:26.614334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:26.614338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:26.614561 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:26.615092 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:26.615104 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:26.615107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:26.615419 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:26.615606 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:26.616337 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 19:17:26.616591 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":402,"thread_start_us":299,"threads_started":1}
I20260924 19:17:26.618103 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:26.598163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42201 (local address 127.11.18.65:44441)
0924 19:17:26.598911 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:26.598956 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:26.600696 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:26.601496 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:26.601504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:26.602054 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:26.602628 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:26.602646 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:26.603795 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:26.603803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:26.605273 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:26.605287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:26.606153 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:26.606162 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:26.606410 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:26.607142 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:26.607172 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:26.609848 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:26.613594 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:26.613606 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:26.613627 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:26.614161 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:26.614716 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:26.614722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:26.614725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:26.614959 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:26.615564 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:26.615573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:26.615951 (+ 378us) client_negotiation.cc:769] Sending connection context
0924 19:17:26.616231 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 19:17:26.616838 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":299,"threads_started":1}
I20260924 19:17:26.620160 28724 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42201
I20260924 19:17:26.620507 28724 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:26.621251 28724 heartbeater.cc:511] Master 127.11.18.126:42201 requested a full tablet report, sending...
I20260924 19:17:26.623781 28535 ts_manager.cc:194] Registered new tserver with Master: cb214bd1463c479eba5433b88144000d (127.11.18.65:38369)
I20260924 19:17:26.626369 28535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:44441
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:17:26.667984 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42201
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:27.010274 28730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:27.010649 28730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:27.010846 28730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:27.019141 28730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:27.019305 28730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:27.019492 28730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:17:27.029655 28730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:27.030972 28730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:27.032363 28730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:27.046221 28736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:27.046254 28738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:27.046341 28735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:27.054350 28730 server_base.cc:1034] running on GCE node
I20260924 19:17:27.055094 28730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:27.056278 28730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:27.057782 28730 hybrid_clock.cc:648] HybridClock initialized: now 1790277447057711 us; error 67 us; skew 500 ppm
I20260924 19:17:27.058332 28730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:27 dist-test-slave-0lgr krb5kdc[28490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277447, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:27.062708 28730 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:17:27.064749 28730 webserver.cc:460] Webserver started at http://127.11.18.66:35349/ using document root <none> and password file <none>
I20260924 19:17:27.065672 28730 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:27.065845 28730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:27.066231 28730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:27.069630 28730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "a2fb51e9765942da986a82248a32c205"
format_stamp: "Formatted at 2026-09-24 19:17:27 on dist-test-slave-0lgr"
server_key: "08b9fc109095c98f48619a0b13fb6a6e"
server_key_iv: "b4ebd4c25202d78537d30c08a8f49233"
server_key_version: "encryptionkey@0"
I20260924 19:17:27.070503 28730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "a2fb51e9765942da986a82248a32c205"
format_stamp: "Formatted at 2026-09-24 19:17:27 on dist-test-slave-0lgr"
server_key: "08b9fc109095c98f48619a0b13fb6a6e"
server_key_iv: "b4ebd4c25202d78537d30c08a8f49233"
server_key_version: "encryptionkey@0"
I20260924 19:17:27.076356 28730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:17:27.080278 28745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:27.081812 28730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:27.082007 28730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "a2fb51e9765942da986a82248a32c205"
format_stamp: "Formatted at 2026-09-24 19:17:27 on dist-test-slave-0lgr"
server_key: "08b9fc109095c98f48619a0b13fb6a6e"
server_key_iv: "b4ebd4c25202d78537d30c08a8f49233"
server_key_version: "encryptionkey@0"
I20260924 19:17:27.082221 28730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:27.115660 28730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:27.120267 28730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:27.120587 28730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:27.121690 28730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:27.123157 28730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:27.123265 28730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:27.123431 28730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:27.123528 28730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 19:17:27.168294 28730 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:32839
I20260924 19:17:27.168402 28857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:32839 every 8 connection(s)
I20260924 19:17:27.169819 28730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:17:27.174877 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 28730
I20260924 19:17:27.175098 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:17:27.175611 11337 external_mini_cluster.cc:1428] Setting key 2293d63ababfe3a5624bb02139d14044
Sep 24 19:17:27 dist-test-slave-0lgr krb5kdc[28490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277447, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:27.195204 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.172984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:56121 (local address 127.11.18.126:42201)
0924 19:17:27.173565 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:27.173573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:27.174554 (+ 981us) server_negotiation.cc:407] Connection header received
0924 19:17:27.177635 (+ 3081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:27.177641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:27.177752 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:27.177888 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:27.179575 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.180425 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.182238 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.182740 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.182882 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.187419 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:27.187429 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:27.187434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:27.187482 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:27.191002 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:27.192126 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:27.192136 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:27.192143 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:27.192325 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:27.192918 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:27.192927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:27.192933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:27.193365 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:27.193534 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:27.194542 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0924 19:17:27.194783 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":255,"threads_started":1}
I20260924 19:17:27.197141 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.173551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42201 (local address 127.11.18.66:56121)
0924 19:17:27.174363 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:27.174430 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:27.177272 (+ 2842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:27.178154 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:27.178163 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:27.178777 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:27.179389 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.179408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.180612 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.180617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.182074 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.182089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.183001 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.183009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.183276 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.184039 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:27.184070 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:27.187077 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:27.191184 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:27.191197 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:27.191222 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:27.191946 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:27.192478 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:27.192487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:27.192495 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:27.192778 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:27.193523 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:27.193535 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:27.194074 (+ 539us) client_negotiation.cc:769] Sending connection context
0924 19:17:27.194426 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 19:17:27.195081 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":313,"threads_started":1}
I20260924 19:17:27.198964 28858 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42201
I20260924 19:17:27.199335 28858 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:27.200332 28858 heartbeater.cc:511] Master 127.11.18.126:42201 requested a full tablet report, sending...
I20260924 19:17:27.202085 28535 ts_manager.cc:194] Registered new tserver with Master: a2fb51e9765942da986a82248a32c205 (127.11.18.66:32839)
I20260924 19:17:27.203158 28535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:56121
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:17:27.245414 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42201
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:27.617542 28864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:27.617849 28864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:27.618005 28864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:27.625483 28864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:27.625598 28864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:27.625820 28864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:17:27.630017 28724 heartbeater.cc:503] Master 127.11.18.126:42201 was elected leader, sending a full tablet report...
I20260924 19:17:27.636281 28864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:27.637671 28864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:27.638998 28864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:27.653119 28869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:27.653004 28870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:27.653597 28872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:27.654853 28864 server_base.cc:1034] running on GCE node
I20260924 19:17:27.655690 28864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:27.656855 28864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:27.658129 28864 hybrid_clock.cc:648] HybridClock initialized: now 1790277447658088 us; error 37 us; skew 500 ppm
I20260924 19:17:27.658548 28864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:27 dist-test-slave-0lgr krb5kdc[28490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277447, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:27.662560 28864 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:17:27.664937 28864 webserver.cc:460] Webserver started at http://127.11.18.67:34841/ using document root <none> and password file <none>
I20260924 19:17:27.665787 28864 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:27.665915 28864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:27.666198 28864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:27.669428 28864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "a9af68493428429685a03b622af83830"
format_stamp: "Formatted at 2026-09-24 19:17:27 on dist-test-slave-0lgr"
server_key: "91ac75a9301b94f9b2e23383b3e4d9fc"
server_key_iv: "c92c9019175efaf1a10000a9e833e7dc"
server_key_version: "encryptionkey@0"
I20260924 19:17:27.670245 28864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "a9af68493428429685a03b622af83830"
format_stamp: "Formatted at 2026-09-24 19:17:27 on dist-test-slave-0lgr"
server_key: "91ac75a9301b94f9b2e23383b3e4d9fc"
server_key_iv: "c92c9019175efaf1a10000a9e833e7dc"
server_key_version: "encryptionkey@0"
I20260924 19:17:27.675984 28864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:17:27.679900 28879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:27.681521 28864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:27.681718 28864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "a9af68493428429685a03b622af83830"
format_stamp: "Formatted at 2026-09-24 19:17:27 on dist-test-slave-0lgr"
server_key: "91ac75a9301b94f9b2e23383b3e4d9fc"
server_key_iv: "c92c9019175efaf1a10000a9e833e7dc"
server_key_version: "encryptionkey@0"
I20260924 19:17:27.681952 28864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:27.710691 28864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:27.715315 28864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:27.715646 28864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:27.716742 28864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:27.718235 28864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:27.718340 28864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:27.718493 28864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:27.718567 28864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:27.754742 28864 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:36151
I20260924 19:17:27.754801 28991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:36151 every 8 connection(s)
I20260924 19:17:27.756189 28864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:17:27.757869 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 28864
I20260924 19:17:27.758056 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:17:27.758677 11337 external_mini_cluster.cc:1428] Setting key bb865f831a31bed398c819a999cef3d6
Sep 24 19:17:27 dist-test-slave-0lgr krb5kdc[28490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277447, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:27.777530 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.759203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:37489 (local address 127.11.18.126:42201)
0924 19:17:27.759787 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:27.759793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:27.760841 (+ 1048us) server_negotiation.cc:407] Connection header received
0924 19:17:27.763088 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:27.763093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:27.763223 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:27.763311 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:27.764706 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.765504 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.767126 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.767531 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.767626 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.771092 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:27.771097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:27.771101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:27.771133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:27.773951 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:27.775108 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:27.775113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:27.775115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:27.775236 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:27.775657 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:27.775660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:27.775663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:27.775988 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:27.776108 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:27.776984 (+ 876us) server_negotiation.cc:299] Negotiation successful
0924 19:17:27.777171 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
I20260924 19:17:27.778488 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.759819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:42201 (local address 127.11.18.67:37489)
0924 19:17:27.760660 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:27.760696 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:27.762853 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:27.763502 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:27.763510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:27.764006 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:27.764555 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.764571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.765678 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.765683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.766983 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.767000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.767716 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.767720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.767917 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.768536 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:27.768559 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:27.770884 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:27.774096 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:27.774106 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:27.774135 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:27.774769 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:27.775349 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:27.775353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:27.775355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:27.775546 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:27.776126 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:27.776132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:27.776473 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 19:17:27.776827 (+ 354us) client_negotiation.cc:240] Negotiation successful
0924 19:17:27.777256 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":406,"threads_started":1}
I20260924 19:17:27.780038 28992 heartbeater.cc:347] Connected to a master server at 127.11.18.126:42201
I20260924 19:17:27.780360 28992 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:27.781075 28992 heartbeater.cc:511] Master 127.11.18.126:42201 requested a full tablet report, sending...
I20260924 19:17:27.782533 28535 ts_manager.cc:194] Registered new tserver with Master: a9af68493428429685a03b622af83830 (127.11.18.67:36151)
I20260924 19:17:27.783320 28535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:37489
I20260924 19:17:27.787693 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:17:27.805712 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.792438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.11.18.126:42201)
0924 19:17:27.792614 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:27.792618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:27.793102 (+ 484us) server_negotiation.cc:407] Connection header received
0924 19:17:27.793240 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:27.793244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:27.793393 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:27.793495 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:27.794966 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.795721 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.797109 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.797593 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.797692 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.800215 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:27.800220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:27.800223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:27.800256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:27.802948 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:27.803708 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:27.803712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:27.803714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:27.803858 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:27.804315 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:27.804319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:27.804322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:27.804655 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:27.804855 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:27.805168 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 19:17:27.805470 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:17:27.809957 28535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:17:27.816854 28535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:17:27.843422 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.831878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.11.18.67:36151)
0924 19:17:27.832696 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:27.832709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:27.833447 (+ 738us) server_negotiation.cc:407] Connection header received
0924 19:17:27.833802 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:27.833814 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:27.834158 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:27.834353 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:27.836342 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.838015 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.840382 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.841876 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.842010 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.842269 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:27.842902 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 19:17:27.843075 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":421,"threads_started":1}
I20260924 19:17:27.843531 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.831312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:36151 (local address 127.0.0.1:52948)
0924 19:17:27.833294 (+ 1982us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:27.833341 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:27.833581 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:27.834803 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:27.834814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:27.834871 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:27.835342 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.835357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.838237 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.838242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.840201 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.840216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.842409 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.842416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.842622 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.842716 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 19:17:27.842856 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 19:17:27.843053 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1820,"mutex_wait_us":1,"thread_start_us":4360,"threads_started":1}
I20260924 19:17:27.844431 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.832718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.11.18.66:32839)
0924 19:17:27.836348 (+ 3630us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:27.836357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:27.836387 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:17:27.836487 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:27.836495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:27.836764 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:27.836986 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:27.837720 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.839256 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.841374 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.842790 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.842919 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.843090 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:27.843967 (+ 877us) server_negotiation.cc:299] Negotiation successful
0924 19:17:27.844128 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3459,"thread_start_us":3349,"threads_started":1}
I20260924 19:17:27.844677 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.830476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:38369 (local address 127.0.0.1:46984)
0924 19:17:27.831521 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:27.831573 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:27.831804 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:27.837445 (+ 5641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:27.837450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:27.837478 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:27.837917 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.837930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.839850 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.839859 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.841965 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.841978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.843860 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.843871 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.844064 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.844096 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:17:27.844169 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:17:27.844289 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":915,"threads_started":1}
I20260924 19:17:27.844398 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.832212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:32839 (local address 127.0.0.1:51876)
0924 19:17:27.836003 (+ 3791us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:27.836027 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:27.836152 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:27.836988 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:27.836993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:27.837017 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:27.837487 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.837500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.839426 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.839431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.841183 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:27.841194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.843240 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.843247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:27.843428 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.843469 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:17:27.843924 (+ 455us) client_negotiation.cc:240] Negotiation successful
0924 19:17:27.844054 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3624,"thread_start_us":724,"threads_started":1}
I20260924 19:17:27.845269 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:27.830819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.11.18.65:38369)
0924 19:17:27.836844 (+ 6025us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:27.836850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:27.836871 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:17:27.836974 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:27.836980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:27.837190 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:27.838099 (+ 909us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:27.838329 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.839713 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.842137 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:27.843612 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:27.844519 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:27.844730 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:27.844796 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:17:27.844962 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5873,"thread_start_us":5690,"threads_started":1}
I20260924 19:17:27.848101 28927 tablet_service.cc:1467] Processing CreateTablet for tablet aa6211632102431fa2ac2e7b330f5b8d (DEFAULT_TABLE table=test-table [id=609a896fea554c27886e178b1c94f5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:27.848369 28793 tablet_service.cc:1467] Processing CreateTablet for tablet aa6211632102431fa2ac2e7b330f5b8d (DEFAULT_TABLE table=test-table [id=609a896fea554c27886e178b1c94f5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:27.848650 28659 tablet_service.cc:1467] Processing CreateTablet for tablet aa6211632102431fa2ac2e7b330f5b8d (DEFAULT_TABLE table=test-table [id=609a896fea554c27886e178b1c94f5bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:27.849664 28793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6211632102431fa2ac2e7b330f5b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:27.849714 28927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6211632102431fa2ac2e7b330f5b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:27.849900 28659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6211632102431fa2ac2e7b330f5b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:27.862330 29011 tablet_bootstrap.cc:492] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Bootstrap starting.
I20260924 19:17:27.864431 29012 tablet_bootstrap.cc:492] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Bootstrap starting.
I20260924 19:17:27.865980 29013 tablet_bootstrap.cc:492] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Bootstrap starting.
I20260924 19:17:27.866554 29011 tablet_bootstrap.cc:654] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:27.867923 29011 log.cc:824] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:27.868207 29012 tablet_bootstrap.cc:654] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:27.869011 29013 tablet_bootstrap.cc:654] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:27.869439 29012 log.cc:824] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:27.870239 29013 log.cc:824] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:27.871474 29011 tablet_bootstrap.cc:492] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: No bootstrap required, opened a new log
I20260924 19:17:27.871767 29011 ts_tablet_manager.cc:1397] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 19:17:27.873073 29013 tablet_bootstrap.cc:492] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: No bootstrap required, opened a new log
I20260924 19:17:27.873404 29013 ts_tablet_manager.cc:1397] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 19:17:27.873567 29012 tablet_bootstrap.cc:492] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: No bootstrap required, opened a new log
I20260924 19:17:27.873816 29012 ts_tablet_manager.cc:1397] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 19:17:27.880645 29013 raft_consensus.cc:348] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:27.881140 29013 raft_consensus.cc:374] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:27.881235 29013 raft_consensus.cc:729] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9af68493428429685a03b622af83830, State: Initialized, Role: FOLLOWER
I20260924 19:17:27.882037 29013 consensus_queue.cc:260] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:27.882659 29011 raft_consensus.cc:348] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:27.883104 29011 raft_consensus.cc:374] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:27.883172 28992 heartbeater.cc:503] Master 127.11.18.126:42201 was elected leader, sending a full tablet report...
I20260924 19:17:27.883194 29011 raft_consensus.cc:729] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fb51e9765942da986a82248a32c205, State: Initialized, Role: FOLLOWER
I20260924 19:17:27.883867 29013 ts_tablet_manager.cc:1428] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260924 19:17:27.883881 29011 consensus_queue.cc:260] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:27.884663 29012 raft_consensus.cc:348] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:27.885074 29012 raft_consensus.cc:374] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:27.885172 29012 raft_consensus.cc:729] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb214bd1463c479eba5433b88144000d, State: Initialized, Role: FOLLOWER
I20260924 19:17:27.885771 29011 ts_tablet_manager.cc:1428] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260924 19:17:27.885869 29012 consensus_queue.cc:260] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:27.887749 28858 heartbeater.cc:503] Master 127.11.18.126:42201 was elected leader, sending a full tablet report...
I20260924 19:17:27.888579 29012 ts_tablet_manager.cc:1428] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260924 19:17:27.922254 28859 tablet.cc:2369] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:17:28.008013 28993 tablet.cc:2369] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:17:28.098285 28725 tablet.cc:2369] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:28.191256 29017 raft_consensus.cc:482] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:28.191624 29017 raft_consensus.cc:504] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:28.193373 29017 leader_election.cc:288] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb214bd1463c479eba5433b88144000d (127.11.18.65:38369), a2fb51e9765942da986a82248a32c205 (127.11.18.66:32839)
I20260924 19:17:28.200519 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:28.193747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:37901 (local address 127.11.18.65:38369)
0924 19:17:28.193988 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:28.193993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:28.194016 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:17:28.194198 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:28.194207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:28.194297 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:28.194439 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:28.195178 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.196101 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.198735 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.199590 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.199765 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:28.199853 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:28.200131 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 19:17:28.200265 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:17:28.200721 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:28.194108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:35115 (local address 127.11.18.66:32839)
0924 19:17:28.194347 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:28.194352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:28.194614 (+ 262us) server_negotiation.cc:407] Connection header received
0924 19:17:28.194812 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:28.194817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:28.194902 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:28.195006 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:28.195604 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.196516 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.198966 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.199742 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.199951 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:28.200026 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:28.200328 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 19:17:28.200458 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:17:28.200889 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:28.193580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:38369 (local address 127.11.18.67:37901)
0924 19:17:28.193776 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:28.193815 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:28.194024 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:28.194459 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:28.194465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:28.194498 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:28.194656 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:28.194664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.196253 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.196258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:28.198574 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:28.198590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.199750 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.199756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:28.199955 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:28.199982 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 19:17:28.200104 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 19:17:28.200250 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":94}
I20260924 19:17:28.200918 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:28.193985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:32839 (local address 127.11.18.67:35115)
0924 19:17:28.194527 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:28.194547 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:28.194665 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:28.195020 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:28.195024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:28.195048 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:28.195419 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:28.195432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.196708 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.196713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:28.198826 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:28.198835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.199986 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.199991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:28.200163 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:28.200187 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:17:28.200299 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:17:28.200442 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":21,"thread_start_us":296,"threads_started":1}
I20260924 19:17:28.202060 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6211632102431fa2ac2e7b330f5b8d" candidate_uuid: "a9af68493428429685a03b622af83830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb214bd1463c479eba5433b88144000d" is_pre_election: true
I20260924 19:17:28.202064 28813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6211632102431fa2ac2e7b330f5b8d" candidate_uuid: "a9af68493428429685a03b622af83830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fb51e9765942da986a82248a32c205" is_pre_election: true
I20260924 19:17:28.202662 28813 raft_consensus.cc:2454] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9af68493428429685a03b622af83830 in term 0.
I20260924 19:17:28.202662 28679 raft_consensus.cc:2454] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9af68493428429685a03b622af83830 in term 0.
I20260924 19:17:28.203490 28880 leader_election.cc:302] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2fb51e9765942da986a82248a32c205, a9af68493428429685a03b622af83830; no voters:
I20260924 19:17:28.203949 29017 raft_consensus.cc:2781] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:17:28.204125 29017 raft_consensus.cc:482] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:17:28.204247 29017 raft_consensus.cc:3037] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:28.206722 29017 raft_consensus.cc:504] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:28.207394 29017 leader_election.cc:288] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [CANDIDATE]: Term 1 election: Requested vote from peers cb214bd1463c479eba5433b88144000d (127.11.18.65:38369), a2fb51e9765942da986a82248a32c205 (127.11.18.66:32839)
I20260924 19:17:28.207863 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6211632102431fa2ac2e7b330f5b8d" candidate_uuid: "a9af68493428429685a03b622af83830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb214bd1463c479eba5433b88144000d"
I20260924 19:17:28.207919 28813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6211632102431fa2ac2e7b330f5b8d" candidate_uuid: "a9af68493428429685a03b622af83830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fb51e9765942da986a82248a32c205"
I20260924 19:17:28.208091 28679 raft_consensus.cc:3037] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:28.208141 28813 raft_consensus.cc:3037] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:28.210665 28813 raft_consensus.cc:2454] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9af68493428429685a03b622af83830 in term 1.
I20260924 19:17:28.210831 28679 raft_consensus.cc:2454] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9af68493428429685a03b622af83830 in term 1.
I20260924 19:17:28.211159 28880 leader_election.cc:302] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2fb51e9765942da986a82248a32c205, a9af68493428429685a03b622af83830; no voters:
I20260924 19:17:28.211592 29017 raft_consensus.cc:2781] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:28.212154 29017 raft_consensus.cc:686] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 1 LEADER]: Becoming Leader. State: Replica: a9af68493428429685a03b622af83830, State: Running, Role: LEADER
I20260924 19:17:28.212718 29017 consensus_queue.cc:237] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } }
I20260924 19:17:28.218222 28533 catalog_manager.cc:5495] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9af68493428429685a03b622af83830 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "a9af68493428429685a03b622af83830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9af68493428429685a03b622af83830" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 36151 } health_report { overall_health: HEALTHY } } }
I20260924 19:17:28.299539 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:28.294062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.11.18.67:36151)
0924 19:17:28.294287 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:28.294293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:28.294361 (+ 68us) server_negotiation.cc:407] Connection header received
0924 19:17:28.294554 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:28.294558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:28.294646 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:28.294771 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:17:28.295160 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.295889 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.297431 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.297826 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.297924 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:28.298482 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:17:28.298992 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:17:28.299102 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:28.299219 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 19:17:28.299286 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:17:28.308580 28813 raft_consensus.cc:1261] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 1 FOLLOWER]: Refusing update from remote peer a9af68493428429685a03b622af83830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:28.308688 28679 raft_consensus.cc:1261] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 1 FOLLOWER]: Refusing update from remote peer a9af68493428429685a03b622af83830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:28.309860 29021 consensus_queue.cc:1035] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2fb51e9765942da986a82248a32c205" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 32839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:28.310352 29017 consensus_queue.cc:1035] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb214bd1463c479eba5433b88144000d" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 38369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:28.324204 29024 mvcc.cc:205] Tried to move back new op lower bound from 7332976428260876288 to 7332976427883929600. Current Snapshot: MvccSnapshot[applied={T|T < 7332976428260876288}]
I20260924 19:17:28.342782 28534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:28.343079 28534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:28.347191 28534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d3df507f3824b3cbc39df3bfc1b204d: Sending DeleteTablet for 3 replicas of tablet aa6211632102431fa2ac2e7b330f5b8d
I20260924 19:17:28.348403 28659 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6211632102431fa2ac2e7b330f5b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:28 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:46984
I20260924 19:17:28.348459 28927 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6211632102431fa2ac2e7b330f5b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:28 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:52948
I20260924 19:17:28.349148 28793 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6211632102431fa2ac2e7b330f5b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:28 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:51876
I20260924 19:17:28.351210 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 28596
I20260924 19:17:28.371847 29036 tablet_replica.cc:330] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: stopping tablet replica
I20260924 19:17:28.372951 29036 raft_consensus.cc:2229] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:28.373813 29036 raft_consensus.cc:2258] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:28.374801 29034 tablet_replica.cc:330] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: stopping tablet replica
I20260924 19:17:28.376879 29036 ts_tablet_manager.cc:1905] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:28.377717 29034 raft_consensus.cc:2229] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:28.378508 29034 raft_consensus.cc:2258] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 19:17:28 dist-test-slave-0lgr krb5kdc[28490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:17:28.384874 29035 tablet_replica.cc:330] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: stopping tablet replica
I20260924 19:17:28.385902 29034 ts_tablet_manager.cc:1905] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:28.389528 29035 raft_consensus.cc:2229] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:17:28.390270 29035 raft_consensus.cc:2258] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:28.392307 29035 ts_tablet_manager.cc:1905] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:28.392633 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:28.352983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.11.18.65:38369)
0924 19:17:28.353230 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:28.353240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:28.371404 (+ 18164us) server_negotiation.cc:407] Connection header received
0924 19:17:28.371880 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:28.371886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:28.371985 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:28.372083 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:28.374088 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.375346 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.377534 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.378250 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.378357 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:28.384226 (+ 5869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:28.384244 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:28.384263 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:28.384321 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:28.388664 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:28.389755 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:28.389765 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:28.389771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:28.389964 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:28.390538 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:28.390544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:28.390547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:28.391036 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:28.391340 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:28.391924 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 19:17:28.392157 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:17:28.392907 29034 ts_tablet_manager.cc:1918] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:28.393077 29034 log.cc:1196] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/aa6211632102431fa2ac2e7b330f5b8d
I20260924 19:17:28.393615 29034 ts_tablet_manager.cc:1939] T aa6211632102431fa2ac2e7b330f5b8d P cb214bd1463c479eba5433b88144000d: Deleting consensus metadata
I20260924 19:17:28.394701 28719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:28.398161 29036 ts_tablet_manager.cc:1918] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:28.398350 29036 log.cc:1196] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/aa6211632102431fa2ac2e7b330f5b8d
I20260924 19:17:28.398890 29036 ts_tablet_manager.cc:1939] T aa6211632102431fa2ac2e7b330f5b8d P a2fb51e9765942da986a82248a32c205: Deleting consensus metadata
I20260924 19:17:28.400929 28519 catalog_manager.cc:4841] TS a2fb51e9765942da986a82248a32c205 (127.11.18.66:32839): tablet aa6211632102431fa2ac2e7b330f5b8d (table test-table [id=609a896fea554c27886e178b1c94f5bf]) successfully deleted
I20260924 19:17:28.404346 29035 ts_tablet_manager.cc:1918] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:28.404536 29035 log.cc:1196] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/aa6211632102431fa2ac2e7b330f5b8d
I20260924 19:17:28.405103 29035 ts_tablet_manager.cc:1939] T aa6211632102431fa2ac2e7b330f5b8d P a9af68493428429685a03b622af83830: Deleting consensus metadata
I20260924 19:17:28.406723 28520 catalog_manager.cc:4841] TS a9af68493428429685a03b622af83830 (127.11.18.67:36151): tablet aa6211632102431fa2ac2e7b330f5b8d (table test-table [id=609a896fea554c27886e178b1c94f5bf]) successfully deleted
I20260924 19:17:28.528311 28519 catalog_manager.cc:4841] TS cb214bd1463c479eba5433b88144000d (127.11.18.65:38369): tablet aa6211632102431fa2ac2e7b330f5b8d (table test-table [id=609a896fea554c27886e178b1c94f5bf]) successfully deleted
I20260924 19:17:28.528733 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 28596
I20260924 19:17:28.553874 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 28730
Sep 24 19:17:28 dist-test-slave-0lgr krb5kdc[28490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:17:28.572288 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:28.554850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.11.18.66:32839)
0924 19:17:28.555148 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:28.555154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:28.555176 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:17:28.555364 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:28.555377 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:28.555461 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:28.555579 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:28.556982 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.557884 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.559478 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.560066 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.560213 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:28.565719 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:28.565736 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:28.565754 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:28.565801 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:28.568856 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:28.569888 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:28.569897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:28.569899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:28.570048 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:28.570514 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:28.570518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:28.570521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:28.570973 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:28.571220 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:28.571677 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 19:17:28.571932 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 19:17:28.573012 28853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:28.667364 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 28730
I20260924 19:17:28.686980 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 28864
Sep 24 19:17:28 dist-test-slave-0lgr krb5kdc[28490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:17:28.711568 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:28.687942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.11.18.67:36151)
0924 19:17:28.688133 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:28.688141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:28.688165 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:17:28.688359 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:28.688365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:28.688454 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:28.688582 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:28.690164 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.691432 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.692995 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:28.693743 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:28.693885 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:28.704467 (+ 10582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:28.704481 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:28.704496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:28.704544 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:28.708006 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:28.709009 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:28.709017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:28.709019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:28.709215 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:28.709756 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:28.709760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:28.709763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:28.710178 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:28.710473 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:28.710866 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 19:17:28.711069 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:17:28.712423 28987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:28.838169 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 28864
I20260924 19:17:28.858534 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 28503
I20260924 19:17:28.859793 28565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:28.979802 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 28503
2026-09-24T19:17:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5044 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T19:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:29Z Disabled control of system clock
I20260924 19:17:29.064546 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:40021
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:40021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:17:29.415360 29045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:29.415669 29045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:29.415819 29045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:29.423480 29045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:29.423601 29045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:29.423673 29045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:29.423787 29045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:29.423863 29045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:29.433362 29045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:40021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:40021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:29.434728 29045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:29.436592 29045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:29.450078 29050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:29.450134 29051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:29.450143 29053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:29.450695 29045 server_base.cc:1034] running on GCE node
I20260924 19:17:29.451570 29045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:29.453167 29045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:29.454373 29045 hybrid_clock.cc:648] HybridClock initialized: now 1790277449454335 us; error 66 us; skew 500 ppm
I20260924 19:17:29.454689 29045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:29.457043 29045 webserver.cc:460] Webserver started at http://127.11.18.126:38627/ using document root <none> and password file <none>
I20260924 19:17:29.457803 29045 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:29.457901 29045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:29.458235 29045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:29.461133 29045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "c664213f05c04470a55947603cea037a"
format_stamp: "Formatted at 2026-09-24 19:17:29 on dist-test-slave-0lgr"
server_key: "7f5b56fbf499315ff6c0ac963a92065d"
server_key_iv: "7415c46dba866a205f0795851d592f5b"
server_key_version: "encryptionkey@0"
I20260924 19:17:29.462029 29045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "c664213f05c04470a55947603cea037a"
format_stamp: "Formatted at 2026-09-24 19:17:29 on dist-test-slave-0lgr"
server_key: "7f5b56fbf499315ff6c0ac963a92065d"
server_key_iv: "7415c46dba866a205f0795851d592f5b"
server_key_version: "encryptionkey@0"
I20260924 19:17:29.467262 29045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:17:29.471109 29059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:29.472674 29045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:17:29.472883 29045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "c664213f05c04470a55947603cea037a"
format_stamp: "Formatted at 2026-09-24 19:17:29 on dist-test-slave-0lgr"
server_key: "7f5b56fbf499315ff6c0ac963a92065d"
server_key_iv: "7415c46dba866a205f0795851d592f5b"
server_key_version: "encryptionkey@0"
I20260924 19:17:29.473150 29045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:29.490669 29045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:29.491592 29045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:29.491911 29045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:29.512609 29045 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:40021
I20260924 19:17:29.512630 29110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:40021 every 8 connection(s)
I20260924 19:17:29.514038 29045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:29.517972 29111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:29.519729 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 29045
I20260924 19:17:29.519912 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:29.520468 11337 external_mini_cluster.cc:1428] Setting key 55717cd1deb31b75dcea86bc10b82c77
I20260924 19:17:29.530699 29111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a: Bootstrap starting.
I20260924 19:17:29.534489 29111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:29.535084 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:29.522493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.11.18.126:40021)
0924 19:17:29.523552 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:29.523563 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:29.523607 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:17:29.525296 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:29.525374 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:29.525849 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:29.526183 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:29.526923 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:29.528314 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:29.530264 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:29.531095 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:29.531231 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:29.532541 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:29.532549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:29.532565 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:29.533185 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:29.533278 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:29.533300 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:29.533489 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:29.533720 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 19:17:29.534040 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":411,"threads_started":1}
I20260924 19:17:29.535674 29111 log.cc:824] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:29.539132 29111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a: No bootstrap required, opened a new log
I20260924 19:17:29.547425 29111 raft_consensus.cc:348] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c664213f05c04470a55947603cea037a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40021 } }
I20260924 19:17:29.547842 29111 raft_consensus.cc:374] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:29.547953 29111 raft_consensus.cc:729] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c664213f05c04470a55947603cea037a, State: Initialized, Role: FOLLOWER
I20260924 19:17:29.548496 29111 consensus_queue.cc:260] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c664213f05c04470a55947603cea037a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40021 } }
I20260924 19:17:29.548745 29111 raft_consensus.cc:388] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:29.548890 29111 raft_consensus.cc:482] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:29.549067 29111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:29.551445 29111 raft_consensus.cc:504] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c664213f05c04470a55947603cea037a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40021 } }
I20260924 19:17:29.552042 29111 leader_election.cc:302] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c664213f05c04470a55947603cea037a; no voters:
I20260924 19:17:29.552826 29111 leader_election.cc:288] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:29.553028 29116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:29.554257 29116 raft_consensus.cc:686] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [term 1 LEADER]: Becoming Leader. State: Replica: c664213f05c04470a55947603cea037a, State: Running, Role: LEADER
I20260924 19:17:29.554854 29116 consensus_queue.cc:237] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c664213f05c04470a55947603cea037a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40021 } }
I20260924 19:17:29.555297 29111 sys_catalog.cc:564] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:29.558319 29118 sys_catalog.cc:455] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c664213f05c04470a55947603cea037a. Latest consensus state: current_term: 1 leader_uuid: "c664213f05c04470a55947603cea037a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c664213f05c04470a55947603cea037a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40021 } } }
I20260924 19:17:29.558823 29118 sys_catalog.cc:458] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:29.558360 29117 sys_catalog.cc:455] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c664213f05c04470a55947603cea037a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c664213f05c04470a55947603cea037a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 40021 } } }
I20260924 19:17:29.559461 29117 sys_catalog.cc:458] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:29.560099 29124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:29.566262 29124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:29.577023 29124 catalog_manager.cc:1347] Generated new cluster ID: a9cfc2e9aa5f43d982a93b389f857e93
I20260924 19:17:29.577163 29124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:29.595053 29124 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:29.596939 29124 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:29.605917 29124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a: Generated new TSK 0
I20260924 19:17:29.607016 29124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:29.620137 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40021
--builtin_ntp_servers=127.11.18.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:17:29.967691 29135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:29.968036 29135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:29.968201 29135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:29.975673 29135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:29.975813 29135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:29.976032 29135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:17:29.985648 29135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:29.987360 29135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:29.989286 29135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:30.004269 29143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:30.004452 29140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:30.004225 29141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:30.005384 29135 server_base.cc:1034] running on GCE node
I20260924 19:17:30.006018 29135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:30.007051 29135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:30.008353 29135 hybrid_clock.cc:648] HybridClock initialized: now 1790277450008272 us; error 66 us; skew 500 ppm
I20260924 19:17:30.008765 29135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:30.011631 29135 webserver.cc:460] Webserver started at http://127.11.18.65:43301/ using document root <none> and password file <none>
I20260924 19:17:30.012432 29135 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:30.012583 29135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:30.012933 29135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:30.016216 29135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "4df3909311504a089a03397ba017fe76"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "dab9497d969d505a1bdffebf15b53d59"
server_key_iv: "fb8f5eea860c89d2e09a9ae44b5b81f4"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.017120 29135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "4df3909311504a089a03397ba017fe76"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "dab9497d969d505a1bdffebf15b53d59"
server_key_iv: "fb8f5eea860c89d2e09a9ae44b5b81f4"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.022912 29135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:17:30.027145 29149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:30.028872 29135 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:17:30.029076 29135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "4df3909311504a089a03397ba017fe76"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "dab9497d969d505a1bdffebf15b53d59"
server_key_iv: "fb8f5eea860c89d2e09a9ae44b5b81f4"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.029320 29135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:30.046075 29135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:30.046975 29135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:30.047308 29135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:30.048357 29135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:30.050045 29135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:30.050170 29135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:30.050313 29135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:30.050405 29135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:30.087087 29135 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:43833
I20260924 19:17:30.087179 29261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:43833 every 8 connection(s)
I20260924 19:17:30.088457 29135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:17:30.096982 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 29135
I20260924 19:17:30.097191 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:17:30.097715 11337 external_mini_cluster.cc:1428] Setting key f0936357bcb77a7031f5d4953f9f1773
I20260924 19:17:30.100472 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40021
--builtin_ntp_servers=127.11.18.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:17:30.102926 29264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:30.091242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:54443 (local address 127.11.18.126:40021)
0924 19:17:30.091910 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:30.091915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:30.093098 (+ 1183us) server_negotiation.cc:407] Connection header received
0924 19:17:30.095219 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:30.095225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:30.095342 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:30.095475 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:30.096662 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:30.097688 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:30.099651 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:30.100112 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:30.100266 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:30.101967 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:30.101973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:30.101976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:30.101997 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:30.102041 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:30.102054 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:30.102161 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:30.102413 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 19:17:30.102524 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":278,"threads_started":1}
I20260924 19:17:30.104090 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:30.091893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:40021 (local address 127.11.18.65:54443)
0924 19:17:30.092915 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:30.092958 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:30.094905 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:30.095732 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:30.095742 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:30.095860 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:30.096462 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:30.096480 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:30.097868 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:30.097873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:30.099505 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:30.099519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:30.100394 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:30.100402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:30.100722 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:30.101591 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:30.101594 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:30.101638 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:30.101729 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:30.102160 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:30.102176 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:30.102187 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:17:30.102402 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 19:17:30.102857 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":351,"threads_started":1}
I20260924 19:17:30.106078 29262 heartbeater.cc:347] Connected to a master server at 127.11.18.126:40021
I20260924 19:17:30.106458 29262 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:30.107419 29262 heartbeater.cc:511] Master 127.11.18.126:40021 requested a full tablet report, sending...
I20260924 19:17:30.109982 29076 ts_manager.cc:194] Registered new tserver with Master: 4df3909311504a089a03397ba017fe76 (127.11.18.65:43833)
I20260924 19:17:30.111474 29076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.65:54443
W20260924 19:17:30.448632 29266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:30.448937 29266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:30.449079 29266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:30.457031 29266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:30.457140 29266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:30.457305 29266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:17:30.467118 29266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:30.468451 29266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:30.470224 29266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:30.483856 29271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:30.483807 29274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:30.483834 29272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:30.484830 29266 server_base.cc:1034] running on GCE node
I20260924 19:17:30.485549 29266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:30.486613 29266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:30.487870 29266 hybrid_clock.cc:648] HybridClock initialized: now 1790277450487829 us; error 34 us; skew 500 ppm
I20260924 19:17:30.488226 29266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:30.491146 29266 webserver.cc:460] Webserver started at http://127.11.18.66:44549/ using document root <none> and password file <none>
I20260924 19:17:30.491952 29266 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:30.492105 29266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:30.492463 29266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:30.495728 29266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "956e63df10f1472f9274436a690a9116"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "337b28229c3805d32f35304f89f822bc"
server_key_iv: "263e953a9f8e82fe52ce1ae9425d8eb1"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.496785 29266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "956e63df10f1472f9274436a690a9116"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "337b28229c3805d32f35304f89f822bc"
server_key_iv: "263e953a9f8e82fe52ce1ae9425d8eb1"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.502524 29266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:17:30.506191 29280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:30.507699 29266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:17:30.507884 29266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "956e63df10f1472f9274436a690a9116"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "337b28229c3805d32f35304f89f822bc"
server_key_iv: "263e953a9f8e82fe52ce1ae9425d8eb1"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.508117 29266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:30.536664 29266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:30.537848 29266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:30.538162 29266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:30.539227 29266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:30.540830 29266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:30.540959 29266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:30.541100 29266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:30.541188 29266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:30.577299 29266 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:40155
I20260924 19:17:30.577481 29392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:40155 every 8 connection(s)
I20260924 19:17:30.578809 29266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:17:30.588423 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 29266
I20260924 19:17:30.588651 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:17:30.589433 11337 external_mini_cluster.cc:1428] Setting key 19510208b6122ff9051f1a65a3d20896
I20260924 19:17:30.593684 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40021
--builtin_ntp_servers=127.11.18.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:17:30.593748 29264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:30.581819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:44753 (local address 127.11.18.126:40021)
0924 19:17:30.582059 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:30.582066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:30.583685 (+ 1619us) server_negotiation.cc:407] Connection header received
0924 19:17:30.585532 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:30.585536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:30.585610 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:30.585709 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:30.586906 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:30.587625 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:30.589684 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:30.590612 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:30.590771 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:30.592755 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:30.592762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:30.592767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:30.592789 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:30.592845 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:30.592864 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:30.593038 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:30.593268 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 19:17:30.593413 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:17:30.594892 29395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:30.582515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:40021 (local address 127.11.18.66:44753)
0924 19:17:30.583492 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:30.583533 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:30.585183 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:30.585988 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:30.585996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:30.586116 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:30.586685 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:30.586702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:30.587789 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:30.587793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:30.589447 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:30.589463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:30.590880 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:30.590886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:30.591256 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:30.592333 (+ 1077us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:30.592336 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:30.592412 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:30.592527 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:30.593039 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:30.593048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:30.593054 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:17:30.593227 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 19:17:30.593645 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":321,"threads_started":1}
I20260924 19:17:30.596482 29393 heartbeater.cc:347] Connected to a master server at 127.11.18.126:40021
I20260924 19:17:30.596827 29393 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:30.597793 29393 heartbeater.cc:511] Master 127.11.18.126:40021 requested a full tablet report, sending...
I20260924 19:17:30.599519 29076 ts_manager.cc:194] Registered new tserver with Master: 956e63df10f1472f9274436a690a9116 (127.11.18.66:40155)
I20260924 19:17:30.600448 29076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.66:44753
W20260924 19:17:30.931509 29396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:30.931819 29396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:30.931926 29396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:30.940588 29396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:30.940717 29396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:30.940923 29396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:17:30.951699 29396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:40021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:30.953101 29396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:30.955091 29396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:30.969852 29401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:30.969942 29404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:30.969863 29402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:30.970685 29396 server_base.cc:1034] running on GCE node
I20260924 19:17:30.971315 29396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:30.972412 29396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:30.973680 29396 hybrid_clock.cc:648] HybridClock initialized: now 1790277450973610 us; error 60 us; skew 500 ppm
I20260924 19:17:30.974114 29396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:30.976802 29396 webserver.cc:460] Webserver started at http://127.11.18.67:39157/ using document root <none> and password file <none>
I20260924 19:17:30.977663 29396 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:30.977809 29396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:30.978128 29396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:30.981177 29396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "d5295e3ec06243df96499b1e13e08a6b"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "940d5e67cfddc5f7d0003a67b554962d"
server_key_iv: "51bbc1ccc5cf7812926e3cfbaef18c55"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.982048 29396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "d5295e3ec06243df96499b1e13e08a6b"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "940d5e67cfddc5f7d0003a67b554962d"
server_key_iv: "51bbc1ccc5cf7812926e3cfbaef18c55"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.987356 29396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:17:30.991710 29410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:30.993269 29396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:17:30.993484 29396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "d5295e3ec06243df96499b1e13e08a6b"
format_stamp: "Formatted at 2026-09-24 19:17:30 on dist-test-slave-0lgr"
server_key: "940d5e67cfddc5f7d0003a67b554962d"
server_key_iv: "51bbc1ccc5cf7812926e3cfbaef18c55"
server_key_version: "encryptionkey@0"
I20260924 19:17:30.993732 29396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:31.009303 29396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:31.010268 29396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:31.010579 29396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:31.011649 29396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:31.013427 29396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:31.013551 29396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:31.013696 29396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:31.013787 29396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:31.050318 29396 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:46739
I20260924 19:17:31.050402 29522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:46739 every 8 connection(s)
I20260924 19:17:31.051863 29396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:17:31.061115 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 29396
I20260924 19:17:31.061373 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:17:31.061883 11337 external_mini_cluster.cc:1428] Setting key be27744de5f7efddfa2a104d9f7ebc07
I20260924 19:17:31.066736 29264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.054761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:56743 (local address 127.11.18.126:40021)
0924 19:17:31.054980 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.054985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.056708 (+ 1723us) server_negotiation.cc:407] Connection header received
0924 19:17:31.058619 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.058624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.058700 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.058806 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:31.060001 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.061119 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.063332 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.064001 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.064144 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.065798 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:31.065804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:31.065809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:31.065829 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:31.065863 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:31.065876 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:31.065959 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.066385 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.066480 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:17:31.067695 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.055483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:40021 (local address 127.11.18.67:56743)
0924 19:17:31.056475 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.056516 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:31.058327 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:31.059072 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:31.059081 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:31.059180 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:31.059811 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.059830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.061390 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.061395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.063123 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.063147 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.064254 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.064262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.064562 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.065469 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:31.065472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:31.065515 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:31.065594 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:31.066004 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:31.066012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:31.066018 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:17:31.066321 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 19:17:31.066634 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":324,"threads_started":1}
I20260924 19:17:31.069211 29523 heartbeater.cc:347] Connected to a master server at 127.11.18.126:40021
I20260924 19:17:31.069541 29523 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:31.070312 29523 heartbeater.cc:511] Master 127.11.18.126:40021 requested a full tablet report, sending...
I20260924 19:17:31.071851 29076 ts_manager.cc:194] Registered new tserver with Master: d5295e3ec06243df96499b1e13e08a6b (127.11.18.67:46739)
I20260924 19:17:31.072649 29076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.67:56743
I20260924 19:17:31.078235 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:17:31.090488 29264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.083249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.11.18.126:40021)
0924 19:17:31.083518 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.083524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.084015 (+ 491us) server_negotiation.cc:407] Connection header received
0924 19:17:31.084152 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.084156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.084241 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.084342 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:31.084960 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.085782 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.087702 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.088393 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.088594 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.089648 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:31.089655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:31.089660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:31.089684 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:31.089730 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:31.089749 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:31.089858 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.090061 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.090164 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 19:17:31.095148 29076 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:17:31.103015 29076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:17:31.115447 29262 heartbeater.cc:503] Master 127.11.18.126:40021 was elected leader, sending a full tablet report...
I20260924 19:17:31.132797 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.117188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46739 (local address 127.0.0.1:35168)
0924 19:17:31.118405 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.118474 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:31.118726 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:31.121546 (+ 2820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:31.121558 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:31.121602 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:31.122166 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.122183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.125173 (+ 2990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.125182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.128597 (+ 3415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.128621 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.131410 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.131421 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.131764 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.131847 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 19:17:31.131992 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 19:17:31.132271 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":625,"threads_started":1}
I20260924 19:17:31.135186 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.117527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.11.18.67:46739)
0924 19:17:31.120671 (+ 3144us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.120685 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.120744 (+ 59us) server_negotiation.cc:407] Connection header received
0924 19:17:31.120879 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.120890 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.121214 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.122453 (+ 1239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:31.122810 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.124965 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.128971 (+ 4006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.131094 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.133819 (+ 2725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.134333 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.134437 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.134699 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2966,"thread_start_us":2946,"threads_started":1}
I20260924 19:17:31.139837 29458 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed59a8857db4be59e6bae08dac92145 (DEFAULT_TABLE table=test-table [id=f93bf124923249b5b3fccf3f95379c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:31.141709 29458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed59a8857db4be59e6bae08dac92145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:31.144919 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.118167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:40155 (local address 127.0.0.1:43644)
0924 19:17:31.133410 (+ 15243us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.133436 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:31.133580 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:31.137356 (+ 3776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:31.137362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:31.137393 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:31.137590 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.137617 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.139850 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.139856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.141976 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.141998 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.144015 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.144021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.144227 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.144265 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:17:31.144358 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 19:17:31.144522 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15053,"mutex_wait_us":35,"thread_start_us":18653,"threads_started":1}
I20260924 19:17:31.145035 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.119073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.11.18.66:40155)
0924 19:17:31.136630 (+ 17557us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.136639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.136673 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:17:31.136812 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.136820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.137094 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.137311 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:31.137799 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.139659 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.142201 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.143811 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.143976 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.144241 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.144446 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.144635 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17385,"thread_start_us":17263,"threads_started":1}
I20260924 19:17:31.147169 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.120193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.0.0.1:46238)
0924 19:17:31.135995 (+ 15802us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.136014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:31.136128 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:31.138157 (+ 2029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:31.138162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:31.138190 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:31.138529 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.138540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.140551 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.140557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.144715 (+ 4158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.144731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.146369 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.146377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.146565 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.146597 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:17:31.146694 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:17:31.146851 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15658,"thread_start_us":15482,"threads_started":1}
I20260924 19:17:31.147977 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.120278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.11.18.65:43833)
0924 19:17:31.137495 (+ 17217us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.137504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.137539 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:17:31.137639 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.137648 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.137929 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.138144 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:31.138702 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.140366 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.144975 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.146169 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.146285 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.146501 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.147420 (+ 919us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.147605 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16978,"thread_start_us":17253,"threads_started":1}
I20260924 19:17:31.150348 29328 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed59a8857db4be59e6bae08dac92145 (DEFAULT_TABLE table=test-table [id=f93bf124923249b5b3fccf3f95379c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:31.151662 29328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed59a8857db4be59e6bae08dac92145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:31.155478 29541 tablet_bootstrap.cc:492] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Bootstrap starting.
I20260924 19:17:31.157665 29197 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed59a8857db4be59e6bae08dac92145 (DEFAULT_TABLE table=test-table [id=f93bf124923249b5b3fccf3f95379c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:31.159299 29541 tablet_bootstrap.cc:654] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:31.159818 29197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed59a8857db4be59e6bae08dac92145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:31.160384 29541 log.cc:824] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:31.166004 29541 tablet_bootstrap.cc:492] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: No bootstrap required, opened a new log
I20260924 19:17:31.166275 29541 ts_tablet_manager.cc:1397] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260924 19:17:31.167280 29543 tablet_bootstrap.cc:492] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Bootstrap starting.
I20260924 19:17:31.171072 29543 tablet_bootstrap.cc:654] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:31.172333 29543 log.cc:824] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:31.175252 29545 tablet_bootstrap.cc:492] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Bootstrap starting.
I20260924 19:17:31.175977 29543 tablet_bootstrap.cc:492] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: No bootstrap required, opened a new log
I20260924 19:17:31.176287 29543 ts_tablet_manager.cc:1397] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260924 19:17:31.177169 29541 raft_consensus.cc:348] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.177670 29541 raft_consensus.cc:374] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:31.177790 29541 raft_consensus.cc:729] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5295e3ec06243df96499b1e13e08a6b, State: Initialized, Role: FOLLOWER
I20260924 19:17:31.178421 29541 consensus_queue.cc:260] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.178992 29545 tablet_bootstrap.cc:654] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:31.179589 29523 heartbeater.cc:503] Master 127.11.18.126:40021 was elected leader, sending a full tablet report...
I20260924 19:17:31.180310 29541 ts_tablet_manager.cc:1428] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260924 19:17:31.180311 29545 log.cc:824] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:31.183605 29545 tablet_bootstrap.cc:492] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: No bootstrap required, opened a new log
I20260924 19:17:31.183924 29545 ts_tablet_manager.cc:1397] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 19:17:31.187299 29543 raft_consensus.cc:348] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.187702 29543 raft_consensus.cc:374] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:31.187829 29543 raft_consensus.cc:729] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956e63df10f1472f9274436a690a9116, State: Initialized, Role: FOLLOWER
I20260924 19:17:31.188441 29543 consensus_queue.cc:260] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.189568 29393 heartbeater.cc:503] Master 127.11.18.126:40021 was elected leader, sending a full tablet report...
I20260924 19:17:31.190219 29543 ts_tablet_manager.cc:1428] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 19:17:31.192680 29545 raft_consensus.cc:348] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.193130 29545 raft_consensus.cc:374] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:31.193267 29545 raft_consensus.cc:729] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df3909311504a089a03397ba017fe76, State: Initialized, Role: FOLLOWER
I20260924 19:17:31.193917 29545 consensus_queue.cc:260] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.195497 29545 ts_tablet_manager.cc:1428] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Time spent starting tablet: real 0.011s user 0.001s sys 0.009s
I20260924 19:17:31.246134 29548 raft_consensus.cc:482] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:31.246482 29548 raft_consensus.cc:504] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.248428 29548 leader_election.cc:288] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5295e3ec06243df96499b1e13e08a6b (127.11.18.67:46739), 4df3909311504a089a03397ba017fe76 (127.11.18.65:43833)
I20260924 19:17:31.258473 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.249125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:38673 (local address 127.11.18.67:46739)
0924 19:17:31.249373 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.249380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.250747 (+ 1367us) server_negotiation.cc:407] Connection header received
0924 19:17:31.250935 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.250939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.251000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.251069 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:31.251665 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.252684 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.255962 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.257265 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.257482 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.257553 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.258080 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.258213 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:17:31.258483 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.249142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:52339 (local address 127.11.18.65:43833)
0924 19:17:31.249408 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.249417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.250312 (+ 895us) server_negotiation.cc:407] Connection header received
0924 19:17:31.250572 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.250581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.250652 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.250761 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:31.251384 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.252441 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.255990 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.257275 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.257489 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.257553 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.258080 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.258213 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 19:17:31.258728 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.249528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:46739 (local address 127.11.18.66:38673)
0924 19:17:31.250629 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.250644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:31.250785 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:31.251124 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:31.251128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:31.251160 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:31.251524 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.251532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.252841 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.252848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.255668 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.255679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.257561 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.257568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.257841 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.257931 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 19:17:31.258041 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:17:31.258246 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":217,"thread_start_us":768,"threads_started":1}
I20260924 19:17:31.258739 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.249012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.11.18.66:52339)
0924 19:17:31.250163 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.250181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:31.250403 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:31.250802 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:31.250807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:31.250834 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:31.251226 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.251239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.252603 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.252607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.255661 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:31.255679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.257553 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.257561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:31.257809 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.257927 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 19:17:31.258041 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 19:17:31.258245 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":286,"thread_start_us":523,"threads_started":1}
I20260924 19:17:31.260000 29478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed59a8857db4be59e6bae08dac92145" candidate_uuid: "956e63df10f1472f9274436a690a9116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5295e3ec06243df96499b1e13e08a6b" is_pre_election: true
I20260924 19:17:31.259995 29217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed59a8857db4be59e6bae08dac92145" candidate_uuid: "956e63df10f1472f9274436a690a9116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df3909311504a089a03397ba017fe76" is_pre_election: true
I20260924 19:17:31.260528 29217 raft_consensus.cc:2454] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 956e63df10f1472f9274436a690a9116 in term 0.
I20260924 19:17:31.260535 29478 raft_consensus.cc:2454] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 956e63df10f1472f9274436a690a9116 in term 0.
I20260924 19:17:31.261197 29284 leader_election.cc:302] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 956e63df10f1472f9274436a690a9116, d5295e3ec06243df96499b1e13e08a6b; no voters:
I20260924 19:17:31.261658 29548 raft_consensus.cc:2781] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:17:31.261963 29548 raft_consensus.cc:482] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:17:31.262089 29548 raft_consensus.cc:3037] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:31.264457 29548 raft_consensus.cc:504] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.265172 29548 leader_election.cc:288] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [CANDIDATE]: Term 1 election: Requested vote from peers d5295e3ec06243df96499b1e13e08a6b (127.11.18.67:46739), 4df3909311504a089a03397ba017fe76 (127.11.18.65:43833)
I20260924 19:17:31.265815 29217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed59a8857db4be59e6bae08dac92145" candidate_uuid: "956e63df10f1472f9274436a690a9116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df3909311504a089a03397ba017fe76"
I20260924 19:17:31.265791 29478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed59a8857db4be59e6bae08dac92145" candidate_uuid: "956e63df10f1472f9274436a690a9116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5295e3ec06243df96499b1e13e08a6b"
I20260924 19:17:31.266044 29217 raft_consensus.cc:3037] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:31.266109 29478 raft_consensus.cc:3037] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:31.268347 29217 raft_consensus.cc:2454] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956e63df10f1472f9274436a690a9116 in term 1.
I20260924 19:17:31.268657 29478 raft_consensus.cc:2454] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956e63df10f1472f9274436a690a9116 in term 1.
I20260924 19:17:31.268966 29282 leader_election.cc:302] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df3909311504a089a03397ba017fe76, 956e63df10f1472f9274436a690a9116; no voters:
I20260924 19:17:31.269306 29548 raft_consensus.cc:2781] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:31.270026 29548 raft_consensus.cc:686] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 1 LEADER]: Becoming Leader. State: Replica: 956e63df10f1472f9274436a690a9116, State: Running, Role: LEADER
I20260924 19:17:31.270541 29548 consensus_queue.cc:237] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } }
I20260924 19:17:31.275359 29075 catalog_manager.cc:5495] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 reported cstate change: term changed from 0 to 1, leader changed from <none> to 956e63df10f1472f9274436a690a9116 (127.11.18.66). New cstate: current_term: 1 leader_uuid: "956e63df10f1472f9274436a690a9116" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956e63df10f1472f9274436a690a9116" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 40155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 } health_report { overall_health: UNKNOWN } } }
W20260924 19:17:31.303838 29524 tablet.cc:2369] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:31.324381 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.318550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.11.18.66:40155)
0924 19:17:31.318757 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.318761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.318781 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:17:31.318940 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.318944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.319042 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.319189 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:17:31.319544 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.320389 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.321937 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.322378 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.322480 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.323108 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:17:31.323660 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:17:31.323768 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.324067 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.324148 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 19:17:31.331283 29394 tablet.cc:2369] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:31.332944 29478 raft_consensus.cc:1261] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 1 FOLLOWER]: Refusing update from remote peer 956e63df10f1472f9274436a690a9116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:31.333056 29217 raft_consensus.cc:1261] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 1 FOLLOWER]: Refusing update from remote peer 956e63df10f1472f9274436a690a9116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:31.334165 29548 consensus_queue.cc:1035] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df3909311504a089a03397ba017fe76" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:31.334511 29552 consensus_queue.cc:1035] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5295e3ec06243df96499b1e13e08a6b" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:17:31.341045 29263 tablet.cc:2369] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:31.348351 29557 mvcc.cc:205] Tried to move back new op lower bound from 7332976440649003008 to 7332976440408375296. Current Snapshot: MvccSnapshot[applied={T|T < 7332976440649003008}]
I20260924 19:17:31.368350 29075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:31.368786 29075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:31.372710 29075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c664213f05c04470a55947603cea037a: Sending DeleteTablet for 3 replicas of tablet 8ed59a8857db4be59e6bae08dac92145
I20260924 19:17:31.374116 29458 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed59a8857db4be59e6bae08dac92145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:31 UTC) from {username='slave'} at 127.0.0.1:35168
I20260924 19:17:31.374210 29197 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed59a8857db4be59e6bae08dac92145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:31 UTC) from {username='slave'} at 127.0.0.1:46238
I20260924 19:17:31.375088 29565 tablet_replica.cc:330] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: stopping tablet replica
I20260924 19:17:31.375485 29565 raft_consensus.cc:2229] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:31.375994 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 29135
I20260924 19:17:31.375888 29328 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed59a8857db4be59e6bae08dac92145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:31 UTC) from {username='slave'} at 127.0.0.1:43644
I20260924 19:17:31.376073 29565 raft_consensus.cc:2258] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:31.378433 29565 ts_tablet_manager.cc:1905] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:31.382627 29566 tablet_replica.cc:330] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: stopping tablet replica
I20260924 19:17:31.383613 29566 raft_consensus.cc:2229] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:31.384822 29566 raft_consensus.cc:2258] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:31.388195 29565 ts_tablet_manager.cc:1918] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:31.388449 29565 log.cc:1196] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/8ed59a8857db4be59e6bae08dac92145
I20260924 19:17:31.389087 29565 ts_tablet_manager.cc:1939] T 8ed59a8857db4be59e6bae08dac92145 P d5295e3ec06243df96499b1e13e08a6b: Deleting consensus metadata
I20260924 19:17:31.390002 29566 ts_tablet_manager.cc:1905] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:31.391234 29063 catalog_manager.cc:4841] TS d5295e3ec06243df96499b1e13e08a6b (127.11.18.67:46739): tablet 8ed59a8857db4be59e6bae08dac92145 (table test-table [id=f93bf124923249b5b3fccf3f95379c97]) successfully deleted
I20260924 19:17:31.392282 29567 tablet_replica.cc:330] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: stopping tablet replica
I20260924 19:17:31.392905 29567 raft_consensus.cc:2229] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:17:31.393724 29567 raft_consensus.cc:2258] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:31.396107 29567 ts_tablet_manager.cc:1905] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:31.397207 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.377874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.11.18.65:43833)
0924 19:17:31.378051 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.378059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.388112 (+ 10053us) server_negotiation.cc:407] Connection header received
0924 19:17:31.388685 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.388694 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.388821 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.388912 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:31.389688 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.390745 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.392887 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.393568 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.393668 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.395262 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:31.395274 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:31.395292 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:31.396147 (+ 855us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:31.396209 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:31.396236 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:31.396334 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.396595 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.396766 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:17:31.398373 29566 ts_tablet_manager.cc:1918] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:31.398617 29566 log.cc:1196] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/8ed59a8857db4be59e6bae08dac92145
I20260924 19:17:31.398795 29257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:31.405244 29567 ts_tablet_manager.cc:1918] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:31.405476 29567 log.cc:1196] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/8ed59a8857db4be59e6bae08dac92145
I20260924 19:17:31.406127 29567 ts_tablet_manager.cc:1939] T 8ed59a8857db4be59e6bae08dac92145 P 956e63df10f1472f9274436a690a9116: Deleting consensus metadata
I20260924 19:17:31.407760 29062 catalog_manager.cc:4841] TS 956e63df10f1472f9274436a690a9116 (127.11.18.66:40155): tablet 8ed59a8857db4be59e6bae08dac92145 (table test-table [id=f93bf124923249b5b3fccf3f95379c97]) successfully deleted
I20260924 19:17:31.525820 29566 ts_tablet_manager.cc:1939] T 8ed59a8857db4be59e6bae08dac92145 P 4df3909311504a089a03397ba017fe76: Deleting consensus metadata
I20260924 19:17:31.526952 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 29135
W20260924 19:17:31.546206 29061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.18.65:43833 (error 108)
I20260924 19:17:31.546360 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 29266
I20260924 19:17:31.548305 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.547803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.0.0.1:46256)
0924 19:17:31.548030 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.548168 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 19:17:31.548619 29061 catalog_manager.cc:4584] TS 4df3909311504a089a03397ba017fe76 (127.11.18.65:43833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ed59a8857db4be59e6bae08dac92145: Network error: Client connection negotiation failed: client connection to 127.11.18.65:43833: connect: Connection refused (error 111)
I20260924 19:17:31.555958 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.547454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.11.18.66:40155)
0924 19:17:31.547628 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.547632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.547649 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:17:31.547704 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.547708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.547784 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.547924 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:31.548584 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.549475 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.552039 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.552496 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.552615 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.554142 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:31.554151 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:31.554168 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:31.554914 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:31.554965 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:31.554990 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:31.555090 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.555301 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.555399 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:17:31.556864 29388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:31.602586 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.602023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.0.0.1:46272)
0924 19:17:31.602270 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.602382 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:17:31.603853 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.603419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.0.0.1:46280)
0924 19:17:31.603601 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.603691 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:17:31.664634 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.664074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.0.0.1:46290)
0924 19:17:31.664307 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.664417 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:17:31.666033 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.665447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.0.0.1:46306)
0924 19:17:31.665635 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.665716 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 19:17:31.673411 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 29266
I20260924 19:17:31.693495 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 29396
I20260924 19:17:31.701783 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.694343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.11.18.67:46739)
0924 19:17:31.694581 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:31.694588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:31.694788 (+ 200us) server_negotiation.cc:407] Connection header received
0924 19:17:31.694966 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:31.694979 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:31.695072 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:31.695196 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:31.695712 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.696521 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.698357 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:31.698899 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:31.699059 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:31.700341 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:31.700351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:31.700366 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:31.700950 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:31.701002 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:31.701023 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:31.701150 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:31.701305 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:17:31.701446 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 19:17:31.702600 29518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:31.746977 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.746374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.0.0.1:46308)
0924 19:17:31.746638 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.746753 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 19:17:31.748327 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:31.747863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:43833 (local address 127.0.0.1:46310)
0924 19:17:31.748096 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:31.748185 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 19:17:31.796147 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 29396
I20260924 19:17:31.813074 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 29045
I20260924 19:17:31.814386 29106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:31.926928 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 29045
2026-09-24T19:17:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2937 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:32 dist-test-slave-0lgr krb5kdc[29576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:32 dist-test-slave-0lgr krb5kdc[29576](info): set up 2 sockets
Sep 24 19:17:32 dist-test-slave-0lgr krb5kdc[29576](info): commencing operation
krb5kdc: starting...
W20260924 19:17:33.114547 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.008s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:33 dist-test-slave-0lgr krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:17:33.394353 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:44939
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:44939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:33.747557 29589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:33.747891 29589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:33.747987 29589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:33.755574 29589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:33.755760 29589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:33.755879 29589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:33.755996 29589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:33.756067 29589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:33.766634 29589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:44939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:44939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:33.768043 29589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:33.769456 29589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:33.785822 29595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:33.785769 29594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:33.785885 29597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:33.787514 29589 server_base.cc:1034] running on GCE node
I20260924 19:17:33.788475 29589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:33.790315 29589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:33.791571 29589 hybrid_clock.cc:648] HybridClock initialized: now 1790277453791520 us; error 69 us; skew 500 ppm
I20260924 19:17:33.792039 29589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:33 dist-test-slave-0lgr krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277453, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:33.801520 29589 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:17:33.803329 29589 webserver.cc:460] Webserver started at http://127.11.18.126:38201/ using document root <none> and password file <none>
I20260924 19:17:33.804087 29589 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:33.804242 29589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:33.804572 29589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:33.807735 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "e0e6cb61800b4b49a3c1af192e6140f8"
format_stamp: "Formatted at 2026-09-24 19:17:33 on dist-test-slave-0lgr"
server_key: "381f070b37c25bacd420084174095922"
server_key_iv: "0f9f278565800b599fc31beab9e95eaa"
server_key_version: "encryptionkey@0"
I20260924 19:17:33.808539 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "e0e6cb61800b4b49a3c1af192e6140f8"
format_stamp: "Formatted at 2026-09-24 19:17:33 on dist-test-slave-0lgr"
server_key: "381f070b37c25bacd420084174095922"
server_key_iv: "0f9f278565800b599fc31beab9e95eaa"
server_key_version: "encryptionkey@0"
I20260924 19:17:33.814452 29589 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 19:17:33.818527 29604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:33.820072 29589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:17:33.820293 29589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "e0e6cb61800b4b49a3c1af192e6140f8"
format_stamp: "Formatted at 2026-09-24 19:17:33 on dist-test-slave-0lgr"
server_key: "381f070b37c25bacd420084174095922"
server_key_iv: "0f9f278565800b599fc31beab9e95eaa"
server_key_version: "encryptionkey@0"
I20260924 19:17:33.820540 29589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:33.833539 29589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:33.838284 29589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:33.838600 29589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:33.859318 29589 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:44939
I20260924 19:17:33.859320 29655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:44939 every 8 connection(s)
I20260924 19:17:33.860883 29589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:33.861624 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 29589
I20260924 19:17:33.861862 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:33.862519 11337 external_mini_cluster.cc:1428] Setting key 12352d211de87186fe0a226b5e237308
I20260924 19:17:33.866212 29656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:17:33 dist-test-slave-0lgr krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:33.878188 29656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8: Bootstrap starting.
I20260924 19:17:33.882968 29656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:33.884323 29656 log.cc:824] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:33.888096 29656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8: No bootstrap required, opened a new log
I20260924 19:17:33.888255 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:33.864769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.11.18.126:44939)
0924 19:17:33.865678 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:33.865692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:33.865735 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:17:33.867597 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:33.867622 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:33.868033 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:33.868441 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:33.870170 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:33.871605 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:33.873005 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:33.873773 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:33.873975 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:33.878572 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:33.878586 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:33.878608 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:33.878665 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:33.883139 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:33.883977 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:33.883987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:33.883998 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:33.884218 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:33.884813 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:33.884818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:33.884820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:33.885693 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:33.886044 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:33.886666 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 19:17:33.887033 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":371,"threads_started":1}
I20260924 19:17:33.897056 29656 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e6cb61800b4b49a3c1af192e6140f8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44939 } }
I20260924 19:17:33.897751 29656 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:33.897897 29656 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e6cb61800b4b49a3c1af192e6140f8, State: Initialized, Role: FOLLOWER
I20260924 19:17:33.898552 29656 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e6cb61800b4b49a3c1af192e6140f8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44939 } }
I20260924 19:17:33.898833 29656 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:33.899003 29656 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:33.899201 29656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:33.901754 29656 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e6cb61800b4b49a3c1af192e6140f8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44939 } }
I20260924 19:17:33.902328 29656 leader_election.cc:302] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e6cb61800b4b49a3c1af192e6140f8; no voters:
I20260924 19:17:33.903028 29656 leader_election.cc:288] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:33.903208 29661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:33.904055 29661 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [term 1 LEADER]: Becoming Leader. State: Replica: e0e6cb61800b4b49a3c1af192e6140f8, State: Running, Role: LEADER
I20260924 19:17:33.904747 29661 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e6cb61800b4b49a3c1af192e6140f8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44939 } }
I20260924 19:17:33.905443 29656 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:33.908485 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0e6cb61800b4b49a3c1af192e6140f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e6cb61800b4b49a3c1af192e6140f8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44939 } } }
I20260924 19:17:33.908895 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:33.909403 29663 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e6cb61800b4b49a3c1af192e6140f8. Latest consensus state: current_term: 1 leader_uuid: "e0e6cb61800b4b49a3c1af192e6140f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e6cb61800b4b49a3c1af192e6140f8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44939 } } }
I20260924 19:17:33.909708 29663 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:33.910588 29668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:33.917197 29668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:33.929302 29668 catalog_manager.cc:1347] Generated new cluster ID: b9f1f1279c144addab3465e78708636f
I20260924 19:17:33.929467 29668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:33.950379 29668 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:33.951849 29668 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:33.958636 29668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8: Generated new TSK 0
I20260924 19:17:33.959478 29668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:17:34.033931 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44939
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:34.391038 29682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:34.391396 29682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:34.391578 29682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:34.399804 29682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:34.399984 29682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:34.400170 29682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:17:34.410769 29682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:34.412153 29682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:34.413581 29682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:34.427609 29688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:34.427552 29690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:34.427871 29687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:34.428787 29682 server_base.cc:1034] running on GCE node
I20260924 19:17:34.429638 29682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:34.430724 29682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:34.431989 29682 hybrid_clock.cc:648] HybridClock initialized: now 1790277454431950 us; error 66 us; skew 500 ppm
I20260924 19:17:34.432344 29682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:34 dist-test-slave-0lgr krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:34.436028 29682 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:17:34.437916 29682 webserver.cc:460] Webserver started at http://127.11.18.65:34767/ using document root <none> and password file <none>
I20260924 19:17:34.438689 29682 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:34.438862 29682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:34.439217 29682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:34.442646 29682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "31a86d2834754da3924e7852454c018a"
format_stamp: "Formatted at 2026-09-24 19:17:34 on dist-test-slave-0lgr"
server_key: "b645dbb291a18b1c84d27b556471a41b"
server_key_iv: "8f454b0349947fbcf2ce9925fa2f9a0a"
server_key_version: "encryptionkey@0"
I20260924 19:17:34.443518 29682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "31a86d2834754da3924e7852454c018a"
format_stamp: "Formatted at 2026-09-24 19:17:34 on dist-test-slave-0lgr"
server_key: "b645dbb291a18b1c84d27b556471a41b"
server_key_iv: "8f454b0349947fbcf2ce9925fa2f9a0a"
server_key_version: "encryptionkey@0"
I20260924 19:17:34.449306 29682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:17:34.453301 29697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:34.454779 29682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:34.454977 29682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "31a86d2834754da3924e7852454c018a"
format_stamp: "Formatted at 2026-09-24 19:17:34 on dist-test-slave-0lgr"
server_key: "b645dbb291a18b1c84d27b556471a41b"
server_key_iv: "8f454b0349947fbcf2ce9925fa2f9a0a"
server_key_version: "encryptionkey@0"
I20260924 19:17:34.455204 29682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:34.472429 29682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:34.477203 29682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:34.477546 29682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:34.478585 29682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:34.479997 29682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:34.480103 29682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:34.480264 29682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:34.480338 29682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:34.517369 29682 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:37839
I20260924 19:17:34.517469 29809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:37839 every 8 connection(s)
I20260924 19:17:34.518680 29682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:17:34.523994 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 29682
I20260924 19:17:34.524205 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:17:34.524803 11337 external_mini_cluster.cc:1428] Setting key 9c6ff198bb8ba136aef8517f4e5b8e31
Sep 24 19:17:34 dist-test-slave-0lgr krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:34.543509 29812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:34.521649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:35923 (local address 127.11.18.126:44939)
0924 19:17:34.522222 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:34.522227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:34.523055 (+ 828us) server_negotiation.cc:407] Connection header received
0924 19:17:34.525790 (+ 2735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:34.525796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:34.525900 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:34.526024 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:34.527937 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:34.528664 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:34.531236 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:34.531672 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:34.531788 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:34.536497 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:34.536507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:34.536515 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:34.536569 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:34.539879 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:34.540784 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:34.540790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:34.540793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:34.540928 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:34.541492 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:34.541505 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:34.541508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:34.541824 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:34.542007 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:34.542813 (+ 806us) server_negotiation.cc:299] Negotiation successful
0924 19:17:34.543081 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":332,"threads_started":1}
I20260924 19:17:34.544615 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:34.522098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:44939 (local address 127.11.18.65:35923)
0924 19:17:34.522872 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:34.522917 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:34.525498 (+ 2581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:34.526284 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:34.526292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:34.526882 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:34.527703 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:34.527737 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:34.528846 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:34.528855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:34.531039 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:34.531058 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:34.531930 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:34.531939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:34.532293 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:34.533128 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:34.533163 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:34.536190 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:34.540080 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:34.540094 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:34.540118 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:34.540615 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:34.541062 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:34.541068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:34.541072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:34.541301 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:34.541975 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:34.541984 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:34.542392 (+ 408us) client_negotiation.cc:769] Sending connection context
0924 19:17:34.542690 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 19:17:34.543303 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":313,"threads_started":1}
I20260924 19:17:34.546634 29810 heartbeater.cc:347] Connected to a master server at 127.11.18.126:44939
I20260924 19:17:34.546980 29810 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:34.547760 29810 heartbeater.cc:511] Master 127.11.18.126:44939 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:17:34.550434 29621 ts_manager.cc:194] Registered new tserver with Master: 31a86d2834754da3924e7852454c018a (127.11.18.65:37839)
I20260924 19:17:34.551918 29621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:35923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:17:34.586738 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44939
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:34.933108 29816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:34.933467 29816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:34.933569 29816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:34.941042 29816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:34.941164 29816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:34.941289 29816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:17:34.951499 29816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:34.952930 29816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:34.954454 29816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:34.967871 29821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:34.967878 29824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:34.967918 29822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:34.968575 29816 server_base.cc:1034] running on GCE node
I20260924 19:17:34.969249 29816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:34.970335 29816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:34.971578 29816 hybrid_clock.cc:648] HybridClock initialized: now 1790277454971513 us; error 46 us; skew 500 ppm
I20260924 19:17:34.971940 29816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:34 dist-test-slave-0lgr krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:34.975180 29816 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:17:34.976608 29816 webserver.cc:460] Webserver started at http://127.11.18.66:44389/ using document root <none> and password file <none>
I20260924 19:17:34.977259 29816 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:34.977416 29816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:34.977696 29816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:34.980669 29816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "58f57c1ad50848b5a921725d5e51a09d"
format_stamp: "Formatted at 2026-09-24 19:17:34 on dist-test-slave-0lgr"
server_key: "7f689c1d14b96c21b621b2640e1d8672"
server_key_iv: "8d726b720b734b7c0b3d834c2fe8dd60"
server_key_version: "encryptionkey@0"
I20260924 19:17:34.981437 29816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "58f57c1ad50848b5a921725d5e51a09d"
format_stamp: "Formatted at 2026-09-24 19:17:34 on dist-test-slave-0lgr"
server_key: "7f689c1d14b96c21b621b2640e1d8672"
server_key_iv: "8d726b720b734b7c0b3d834c2fe8dd60"
server_key_version: "encryptionkey@0"
I20260924 19:17:34.986698 29816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:17:34.990346 29831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:34.992005 29816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:17:34.992182 29816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "58f57c1ad50848b5a921725d5e51a09d"
format_stamp: "Formatted at 2026-09-24 19:17:34 on dist-test-slave-0lgr"
server_key: "7f689c1d14b96c21b621b2640e1d8672"
server_key_iv: "8d726b720b734b7c0b3d834c2fe8dd60"
server_key_version: "encryptionkey@0"
I20260924 19:17:34.992386 29816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:35.022416 29816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:35.027159 29816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:35.027465 29816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:35.028522 29816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:35.029973 29816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:35.030045 29816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:35.030181 29816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:35.030241 29816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:35.065795 29816 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:46193
I20260924 19:17:35.065874 29943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:46193 every 8 connection(s)
I20260924 19:17:35.067258 29816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:17:35.073783 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 29816
I20260924 19:17:35.074040 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:17:35.074540 11337 external_mini_cluster.cc:1428] Setting key 5542b6373e93460b9c0b984e2437ac58
Sep 24 19:17:35 dist-test-slave-0lgr krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:35.091651 29946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.069922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:49495 (local address 127.11.18.126:44939)
0924 19:17:35.070389 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.070394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.071238 (+ 844us) server_negotiation.cc:407] Connection header received
0924 19:17:35.073105 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.073112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.073211 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.073336 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:35.075013 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.076031 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.078926 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.079437 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.079557 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.084116 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:35.084128 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:35.084136 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:35.084192 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:35.087686 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:35.088793 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:35.088798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:35.088801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:35.089052 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:35.089647 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:35.089652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:35.089654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:35.090030 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:35.090161 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.091126 (+ 965us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.091333 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":261,"threads_started":1}
I20260924 19:17:35.093070 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.070392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:44939 (local address 127.11.18.66:49495)
0924 19:17:35.071070 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:35.071115 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:35.072828 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:35.073595 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:35.073603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:35.074221 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:35.074837 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.074856 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.076195 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.076200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.078652 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.078673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.079704 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.079714 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.080091 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.080889 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:35.080926 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:35.083849 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:35.087890 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:35.087904 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:35.087926 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:35.088624 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:35.089182 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:35.089190 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:35.089196 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:35.089485 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:35.090174 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:35.090185 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:35.090679 (+ 494us) client_negotiation.cc:769] Sending connection context
0924 19:17:35.090988 (+ 309us) client_negotiation.cc:240] Negotiation successful
0924 19:17:35.091580 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":261,"threads_started":1}
I20260924 19:17:35.094687 29944 heartbeater.cc:347] Connected to a master server at 127.11.18.126:44939
I20260924 19:17:35.095081 29944 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:35.095813 29944 heartbeater.cc:511] Master 127.11.18.126:44939 requested a full tablet report, sending...
I20260924 19:17:35.097532 29621 ts_manager.cc:194] Registered new tserver with Master: 58f57c1ad50848b5a921725d5e51a09d (127.11.18.66:46193)
I20260924 19:17:35.098393 29621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:49495
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:17:35.139189 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44939
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:35.485080 29950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:35.485462 29950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:35.485631 29950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:35.493242 29950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:35.493443 29950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:35.493644 29950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:17:35.504076 29950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:35.505620 29950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:35.507027 29950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:35.521711 29955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:35.521749 29956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:35.522315 29958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:35.523311 29950 server_base.cc:1034] running on GCE node
I20260924 19:17:35.524066 29950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:35.525455 29950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:35.526813 29950 hybrid_clock.cc:648] HybridClock initialized: now 1790277455526726 us; error 55 us; skew 500 ppm
I20260924 19:17:35.527226 29950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:35 dist-test-slave-0lgr krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:35.531050 29950 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:17:35.532835 29950 webserver.cc:460] Webserver started at http://127.11.18.67:46579/ using document root <none> and password file <none>
I20260924 19:17:35.533712 29950 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:35.533859 29950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:35.534221 29950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:35.537986 29950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "59a9ded8d64c4f88a34b284f8f8f7849"
format_stamp: "Formatted at 2026-09-24 19:17:35 on dist-test-slave-0lgr"
server_key: "fafd2d82e08f6254b8d97a2af78b7693"
server_key_iv: "0cc13837e8409d2c94ac78fb7e36b761"
server_key_version: "encryptionkey@0"
I20260924 19:17:35.538816 29950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "59a9ded8d64c4f88a34b284f8f8f7849"
format_stamp: "Formatted at 2026-09-24 19:17:35 on dist-test-slave-0lgr"
server_key: "fafd2d82e08f6254b8d97a2af78b7693"
server_key_iv: "0cc13837e8409d2c94ac78fb7e36b761"
server_key_version: "encryptionkey@0"
I20260924 19:17:35.544641 29950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:17:35.548867 29965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:35.550592 29950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:35.550860 29950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "59a9ded8d64c4f88a34b284f8f8f7849"
format_stamp: "Formatted at 2026-09-24 19:17:35 on dist-test-slave-0lgr"
server_key: "fafd2d82e08f6254b8d97a2af78b7693"
server_key_iv: "0cc13837e8409d2c94ac78fb7e36b761"
server_key_version: "encryptionkey@0"
I20260924 19:17:35.551112 29950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:35.555606 29810 heartbeater.cc:503] Master 127.11.18.126:44939 was elected leader, sending a full tablet report...
I20260924 19:17:35.570807 29950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:35.576242 29950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:35.576542 29950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:35.577669 29950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:35.579160 29950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:35.579241 29950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:35.579353 29950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:35.579411 29950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:35.617594 29950 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:42201
I20260924 19:17:35.617682 30077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:42201 every 8 connection(s)
I20260924 19:17:35.619045 29950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:17:35.627525 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 29950
I20260924 19:17:35.627740 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:17:35.628284 11337 external_mini_cluster.cc:1428] Setting key d0d707a8caa5487e92f35000dda15cb9
Sep 24 19:17:35 dist-test-slave-0lgr krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:35.642396 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.621902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:40635 (local address 127.11.18.126:44939)
0924 19:17:35.622416 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.622420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.623289 (+ 869us) server_negotiation.cc:407] Connection header received
0924 19:17:35.625413 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.625419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.625520 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.625647 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:35.627214 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.628132 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.630154 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.630638 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.630777 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.635489 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:35.635495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:35.635499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:35.635541 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:35.638703 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:35.639629 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:35.639638 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:35.639642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:35.639792 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:35.640296 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:35.640302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:35.640305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:35.640619 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:35.640814 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.641672 (+ 858us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.641938 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":273,"threads_started":1}
I20260924 19:17:35.643671 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.622382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:44939 (local address 127.11.18.67:40635)
0924 19:17:35.623125 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:35.623164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:35.625087 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:35.625899 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:35.625907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:35.626460 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:35.627029 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.627047 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.628311 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.628315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.630000 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.630014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.630879 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.630886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.631135 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.632064 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:35.632095 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:35.635135 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:35.638912 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:35.638924 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:35.638947 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:35.639478 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:35.639923 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:35.639931 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:35.639935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:35.640163 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:35.640768 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:35.640777 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:35.641193 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 19:17:35.641580 (+ 387us) client_negotiation.cc:240] Negotiation successful
0924 19:17:35.642209 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":278,"threads_started":1}
I20260924 19:17:35.645143 30078 heartbeater.cc:347] Connected to a master server at 127.11.18.126:44939
I20260924 19:17:35.645566 30078 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:35.646426 30078 heartbeater.cc:511] Master 127.11.18.126:44939 requested a full tablet report, sending...
I20260924 19:17:35.648101 29620 ts_manager.cc:194] Registered new tserver with Master: 59a9ded8d64c4f88a34b284f8f8f7849 (127.11.18.67:42201)
I20260924 19:17:35.649178 29620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:40635
I20260924 19:17:35.657769 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:17:35.677254 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.662250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.11.18.126:44939)
0924 19:17:35.662529 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.662535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.662972 (+ 437us) server_negotiation.cc:407] Connection header received
0924 19:17:35.663048 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.663052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.663117 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.663207 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:35.664897 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.665799 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.667417 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.667970 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.668092 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.671244 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:35.671249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:35.671253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:35.671291 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:35.674208 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:35.675208 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:35.675212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:35.675214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:35.675335 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:35.675755 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:35.675758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:35.675760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:35.676117 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:35.676314 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.676716 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.676871 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 19:17:35.681711 29620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:17:35.688591 29620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:17:35.727142 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.702515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.11.18.67:42201)
0924 19:17:35.715363 (+ 12848us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.715374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.715461 (+ 87us) server_negotiation.cc:407] Connection header received
0924 19:17:35.715577 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.715584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.715912 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.716135 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:35.717565 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.719784 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.724305 (+ 4521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.725955 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.726092 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.726341 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.726621 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.726793 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12657,"thread_start_us":12495,"threads_started":1}
I20260924 19:17:35.727496 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.702394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:46193 (local address 127.0.0.1:49164)
0924 19:17:35.715688 (+ 13294us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:35.715707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:35.715847 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:35.716786 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:35.716795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:35.716835 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:35.717499 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.717516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.720058 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.720067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.724343 (+ 4276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.724360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.726662 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.726669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.726840 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.726870 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:17:35.726950 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:17:35.727088 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13167,"mutex_wait_us":731,"thread_start_us":11496,"threads_started":1}
I20260924 19:17:35.728330 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.702677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.11.18.66:46193)
0924 19:17:35.714815 (+ 12138us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.714826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.715837 (+ 1011us) server_negotiation.cc:407] Connection header received
0924 19:17:35.715991 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.715998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.716216 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.716432 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:35.717765 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.719844 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.724579 (+ 4735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.726162 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.726305 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.726520 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.727843 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.728045 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11977,"thread_start_us":11889,"threads_started":1}
I20260924 19:17:35.728435 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.702334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:42201 (local address 127.0.0.1:37272)
0924 19:17:35.714319 (+ 11985us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:35.714377 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:35.714651 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:35.716470 (+ 1819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:35.716483 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:35.716538 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:35.717255 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.717275 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.720058 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.720067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.724057 (+ 3990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.724073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.726150 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.726156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.726356 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.726422 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 19:17:35.726563 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 19:17:35.726777 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11696,"mutex_wait_us":154,"thread_start_us":13013,"threads_started":1}
I20260924 19:17:35.732018 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.716180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:37839 (local address 127.0.0.1:59164)
0924 19:17:35.723683 (+ 7503us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:35.723704 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:35.724456 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:35.725295 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:35.725300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:35.725352 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:35.725752 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.725764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.727770 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.727775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.729935 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.729948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.731234 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.731240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.731409 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.731447 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:17:35.731526 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:17:35.731649 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7344,"thread_start_us":7162,"threads_started":1}
I20260924 19:17:35.732980 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.716269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.11.18.65:37839)
0924 19:17:35.724287 (+ 8018us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.724296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.724330 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:17:35.724682 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.724691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.724992 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.725211 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:35.725931 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.727605 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.730095 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.731023 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.731195 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.731448 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.732435 (+ 987us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.732634 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7835,"thread_start_us":8682,"threads_started":1}
I20260924 19:17:35.732647 30013 tablet_service.cc:1467] Processing CreateTablet for tablet f21c2d1a9cc843be824ccc6972f8a3cd (DEFAULT_TABLE table=test-table [id=9c35572ab0c74c6894602cbe3d4fc53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:35.732650 29879 tablet_service.cc:1467] Processing CreateTablet for tablet f21c2d1a9cc843be824ccc6972f8a3cd (DEFAULT_TABLE table=test-table [id=9c35572ab0c74c6894602cbe3d4fc53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:35.734249 29879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21c2d1a9cc843be824ccc6972f8a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:35.734249 30013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21c2d1a9cc843be824ccc6972f8a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:35.736244 29745 tablet_service.cc:1467] Processing CreateTablet for tablet f21c2d1a9cc843be824ccc6972f8a3cd (DEFAULT_TABLE table=test-table [id=9c35572ab0c74c6894602cbe3d4fc53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:35.737697 29745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21c2d1a9cc843be824ccc6972f8a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:35.749748 30097 tablet_bootstrap.cc:492] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: Bootstrap starting.
I20260924 19:17:35.753136 30097 tablet_bootstrap.cc:654] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:35.754154 30098 tablet_bootstrap.cc:492] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Bootstrap starting.
I20260924 19:17:35.754652 30097 log.cc:824] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:35.755069 30099 tablet_bootstrap.cc:492] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Bootstrap starting.
I20260924 19:17:35.759568 30098 tablet_bootstrap.cc:654] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:35.759801 30099 tablet_bootstrap.cc:654] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:35.760382 30097 tablet_bootstrap.cc:492] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: No bootstrap required, opened a new log
I20260924 19:17:35.760686 30097 ts_tablet_manager.cc:1397] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260924 19:17:35.761086 30098 log.cc:824] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:35.761108 30099 log.cc:824] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:35.764493 30098 tablet_bootstrap.cc:492] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: No bootstrap required, opened a new log
I20260924 19:17:35.764514 30099 tablet_bootstrap.cc:492] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: No bootstrap required, opened a new log
I20260924 19:17:35.764770 30099 ts_tablet_manager.cc:1397] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 19:17:35.764854 30098 ts_tablet_manager.cc:1397] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 19:17:35.771916 30097 raft_consensus.cc:348] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
I20260924 19:17:35.772444 30097 raft_consensus.cc:374] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:35.772584 30097 raft_consensus.cc:729] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a86d2834754da3924e7852454c018a, State: Initialized, Role: FOLLOWER
I20260924 19:17:35.773295 30097 consensus_queue.cc:260] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
I20260924 19:17:35.773512 30098 raft_consensus.cc:348] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
I20260924 19:17:35.774070 30098 raft_consensus.cc:374] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:35.774232 30098 raft_consensus.cc:729] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a9ded8d64c4f88a34b284f8f8f7849, State: Initialized, Role: FOLLOWER
I20260924 19:17:35.774976 30098 consensus_queue.cc:260] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
I20260924 19:17:35.775343 30097 ts_tablet_manager.cc:1428] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260924 19:17:35.776355 30078 heartbeater.cc:503] Master 127.11.18.126:44939 was elected leader, sending a full tablet report...
I20260924 19:17:35.777128 30098 ts_tablet_manager.cc:1428] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 19:17:35.778851 30099 raft_consensus.cc:348] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
I20260924 19:17:35.779405 30099 raft_consensus.cc:374] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:35.779521 30099 raft_consensus.cc:729] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f57c1ad50848b5a921725d5e51a09d, State: Initialized, Role: FOLLOWER
I20260924 19:17:35.780210 30099 consensus_queue.cc:260] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
I20260924 19:17:35.781553 29944 heartbeater.cc:503] Master 127.11.18.126:44939 was elected leader, sending a full tablet report...
I20260924 19:17:35.782369 30099 ts_tablet_manager.cc:1428] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20260924 19:17:35.793610 30105 raft_consensus.cc:482] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:35.793901 30105 raft_consensus.cc:504] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
I20260924 19:17:35.795588 30105 leader_election.cc:288] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59a9ded8d64c4f88a34b284f8f8f7849 (127.11.18.67:42201), 31a86d2834754da3924e7852454c018a (127.11.18.65:37839)
I20260924 19:17:35.806859 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.796066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:53069 (local address 127.11.18.67:42201)
0924 19:17:35.796291 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.796297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.797467 (+ 1170us) server_negotiation.cc:407] Connection header received
0924 19:17:35.797617 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.797623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.797725 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.797821 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:35.798842 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.800188 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.803806 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.805276 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.805500 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.805588 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.806391 (+ 803us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.806555 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:17:35.806859 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.796120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:34537 (local address 127.11.18.65:37839)
0924 19:17:35.796315 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.796323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.797403 (+ 1080us) server_negotiation.cc:407] Connection header received
0924 19:17:35.797617 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.797623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.797718 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.797828 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:35.798833 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.800119 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.803812 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.805289 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.805501 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.805597 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.806360 (+ 763us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.806568 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:17:35.807067 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.796126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:37839 (local address 127.11.18.66:34537)
0924 19:17:35.797209 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:35.797232 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:35.797464 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:35.797873 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:35.797879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:35.797941 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:35.798621 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.798642 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.800345 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.800348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.803598 (+ 3250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.803609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.805834 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.805843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.806074 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.806183 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 19:17:35.806323 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 19:17:35.806510 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":404,"spinlock_wait_cycles":13696,"thread_start_us":630,"threads_started":1}
I20260924 19:17:35.807893 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.796130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:42201 (local address 127.11.18.66:53069)
0924 19:17:35.797305 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:35.797354 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:35.797469 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:35.797873 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:35.797879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:35.797938 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:35.798653 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.798662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.800336 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.800343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.803591 (+ 3248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:35.803609 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.805930 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.805936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:35.806192 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.806221 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 19:17:35.806347 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 19:17:35.807543 (+ 1196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":1412,"spinlock_wait_cycles":34048,"thread_start_us":572,"threads_started":1}
I20260924 19:17:35.808400 29765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21c2d1a9cc843be824ccc6972f8a3cd" candidate_uuid: "58f57c1ad50848b5a921725d5e51a09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a86d2834754da3924e7852454c018a" is_pre_election: true
I20260924 19:17:35.809007 29765 raft_consensus.cc:2454] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f57c1ad50848b5a921725d5e51a09d in term 0.
I20260924 19:17:35.808964 30033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21c2d1a9cc843be824ccc6972f8a3cd" candidate_uuid: "58f57c1ad50848b5a921725d5e51a09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" is_pre_election: true
I20260924 19:17:35.809530 30033 raft_consensus.cc:2454] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f57c1ad50848b5a921725d5e51a09d in term 0.
I20260924 19:17:35.809818 29832 leader_election.cc:302] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a86d2834754da3924e7852454c018a, 58f57c1ad50848b5a921725d5e51a09d; no voters:
I20260924 19:17:35.810338 30105 raft_consensus.cc:2781] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:17:35.810501 30105 raft_consensus.cc:482] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:17:35.810634 30105 raft_consensus.cc:3037] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:35.813066 30105 raft_consensus.cc:504] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
I20260924 19:17:35.813822 30105 leader_election.cc:288] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [CANDIDATE]: Term 1 election: Requested vote from peers 59a9ded8d64c4f88a34b284f8f8f7849 (127.11.18.67:42201), 31a86d2834754da3924e7852454c018a (127.11.18.65:37839)
I20260924 19:17:35.814338 29765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21c2d1a9cc843be824ccc6972f8a3cd" candidate_uuid: "58f57c1ad50848b5a921725d5e51a09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a86d2834754da3924e7852454c018a"
I20260924 19:17:35.814337 30033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21c2d1a9cc843be824ccc6972f8a3cd" candidate_uuid: "58f57c1ad50848b5a921725d5e51a09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a9ded8d64c4f88a34b284f8f8f7849"
I20260924 19:17:35.814576 30033 raft_consensus.cc:3037] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:35.814575 29765 raft_consensus.cc:3037] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:35.817138 29765 raft_consensus.cc:2454] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f57c1ad50848b5a921725d5e51a09d in term 1.
I20260924 19:17:35.817175 30033 raft_consensus.cc:2454] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f57c1ad50848b5a921725d5e51a09d in term 1.
I20260924 19:17:35.817631 29832 leader_election.cc:302] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a86d2834754da3924e7852454c018a, 58f57c1ad50848b5a921725d5e51a09d; no voters:
I20260924 19:17:35.817953 30105 raft_consensus.cc:2781] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:35.818450 30105 raft_consensus.cc:686] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 1 LEADER]: Becoming Leader. State: Replica: 58f57c1ad50848b5a921725d5e51a09d, State: Running, Role: LEADER
I20260924 19:17:35.819064 30105 consensus_queue.cc:237] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } }
W20260924 19:17:35.819399 29945 tablet.cc:2369] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:35.824229 29621 catalog_manager.cc:5495] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f57c1ad50848b5a921725d5e51a09d (127.11.18.66). New cstate: current_term: 1 leader_uuid: "58f57c1ad50848b5a921725d5e51a09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f57c1ad50848b5a921725d5e51a09d" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 } health_report { overall_health: UNKNOWN } } }
I20260924 19:17:35.865540 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.859501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.11.18.66:46193)
0924 19:17:35.859729 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.859736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.859902 (+ 166us) server_negotiation.cc:407] Connection header received
0924 19:17:35.860185 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.860189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.860272 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.860435 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:17:35.860887 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.861724 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.863170 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.863596 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.863734 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.864350 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:17:35.864852 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:17:35.865039 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.865166 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.865215 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 19:17:35.870823 30079 tablet.cc:2369] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:35.874755 29765 raft_consensus.cc:1261] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 1 FOLLOWER]: Refusing update from remote peer 58f57c1ad50848b5a921725d5e51a09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:35.875036 30033 raft_consensus.cc:1261] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 1 FOLLOWER]: Refusing update from remote peer 58f57c1ad50848b5a921725d5e51a09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:35.875831 30108 consensus_queue.cc:1035] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a86d2834754da3924e7852454c018a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 37839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:35.876216 30105 consensus_queue.cc:1035] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "59a9ded8d64c4f88a34b284f8f8f7849" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:35.888720 30111 mvcc.cc:205] Tried to move back new op lower bound from 7332976459251335168 to 7332976459039981568. Current Snapshot: MvccSnapshot[applied={T|T < 7332976459251335168}]
I20260924 19:17:35.904258 29621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:35.904558 29621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:35.908721 29621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0e6cb61800b4b49a3c1af192e6140f8: Sending DeleteTablet for 3 replicas of tablet f21c2d1a9cc843be824ccc6972f8a3cd
I20260924 19:17:35.910135 30013 tablet_service.cc:1514] Processing DeleteTablet for tablet f21c2d1a9cc843be824ccc6972f8a3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:35 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:37272
I20260924 19:17:35.910296 29745 tablet_service.cc:1514] Processing DeleteTablet for tablet f21c2d1a9cc843be824ccc6972f8a3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:35 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:59164
I20260924 19:17:35.910375 29879 tablet_service.cc:1514] Processing DeleteTablet for tablet f21c2d1a9cc843be824ccc6972f8a3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:35 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:49164
I20260924 19:17:35.912489 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 29682
I20260924 19:17:35.922688 30121 tablet_replica.cc:330] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: stopping tablet replica
I20260924 19:17:35.923357 30121 raft_consensus.cc:2229] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:35.924119 30121 raft_consensus.cc:2258] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:35.926172 30123 tablet_replica.cc:330] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: stopping tablet replica
I20260924 19:17:35.926736 30123 raft_consensus.cc:2229] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:17:35.927261 30121 ts_tablet_manager.cc:1905] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:35.927513 30123 raft_consensus.cc:2258] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:35.929973 30123 ts_tablet_manager.cc:1905] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:35.935030 30121 ts_tablet_manager.cc:1918] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:35.935285 30121 log.cc:1196] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/f21c2d1a9cc843be824ccc6972f8a3cd
I20260924 19:17:35.935894 30121 ts_tablet_manager.cc:1939] T f21c2d1a9cc843be824ccc6972f8a3cd P 59a9ded8d64c4f88a34b284f8f8f7849: Deleting consensus metadata
I20260924 19:17:35.938079 29606 catalog_manager.cc:4841] TS 59a9ded8d64c4f88a34b284f8f8f7849 (127.11.18.67:42201): tablet f21c2d1a9cc843be824ccc6972f8a3cd (table test-table [id=9c35572ab0c74c6894602cbe3d4fc53e]) successfully deleted
I20260924 19:17:35.938416 30123 ts_tablet_manager.cc:1918] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:35.938671 30123 log.cc:1196] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/f21c2d1a9cc843be824ccc6972f8a3cd
I20260924 19:17:35.939162 30123 ts_tablet_manager.cc:1939] T f21c2d1a9cc843be824ccc6972f8a3cd P 58f57c1ad50848b5a921725d5e51a09d: Deleting consensus metadata
I20260924 19:17:35.940868 29605 catalog_manager.cc:4841] TS 58f57c1ad50848b5a921725d5e51a09d (127.11.18.66:46193): tablet f21c2d1a9cc843be824ccc6972f8a3cd (table test-table [id=9c35572ab0c74c6894602cbe3d4fc53e]) successfully deleted
I20260924 19:17:35.941694 30122 tablet_replica.cc:330] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: stopping tablet replica
I20260924 19:17:35.942210 30122 raft_consensus.cc:2229] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:35.942781 30122 raft_consensus.cc:2258] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:35.944914 30122 ts_tablet_manager.cc:1905] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:17:35 dist-test-slave-0lgr krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:17:35.952178 30122 ts_tablet_manager.cc:1918] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:35.952380 30122 log.cc:1196] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/f21c2d1a9cc843be824ccc6972f8a3cd
I20260924 19:17:35.953023 30122 ts_tablet_manager.cc:1939] T f21c2d1a9cc843be824ccc6972f8a3cd P 31a86d2834754da3924e7852454c018a: Deleting consensus metadata
I20260924 19:17:35.955382 29605 catalog_manager.cc:4841] TS 31a86d2834754da3924e7852454c018a (127.11.18.65:37839): tablet f21c2d1a9cc843be824ccc6972f8a3cd (table test-table [id=9c35572ab0c74c6894602cbe3d4fc53e]) successfully deleted
I20260924 19:17:35.960819 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:35.913615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.11.18.65:37839)
0924 19:17:35.913829 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:35.913836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:35.940013 (+ 26177us) server_negotiation.cc:407] Connection header received
0924 19:17:35.940243 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:35.940249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:35.940340 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:35.940471 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:35.942573 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.944168 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.946115 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:35.946850 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:35.947003 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:35.953039 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:35.953058 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:35.953080 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:35.953139 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:35.957169 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:35.958267 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:35.958278 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:35.958282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:35.958519 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:35.959049 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:35.959053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:35.959055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:35.959547 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:35.959821 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:35.960310 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 19:17:35.960509 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:17:35.961822 29805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:36.056442 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 29682
I20260924 19:17:36.079185 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 29816
Sep 24 19:17:36 dist-test-slave-0lgr krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:17:36.101861 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:36.080182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.11.18.66:46193)
0924 19:17:36.080372 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:36.080378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:36.080440 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:17:36.080734 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:36.080740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:36.080824 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:36.080947 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:36.082671 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:36.084213 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:36.086186 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:36.086993 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:36.087129 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:36.093174 (+ 6045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:36.093189 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:36.093210 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:36.093258 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:36.098447 (+ 5189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:36.099519 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:36.099529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:36.099532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:36.099712 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:36.100234 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:36.100239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:36.100242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:36.100648 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:36.100936 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:36.101379 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 19:17:36.101584 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:17:36.102839 29939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:36.229508 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 29816
I20260924 19:17:36.249704 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 29950
Sep 24 19:17:36 dist-test-slave-0lgr krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:17:36.269637 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:36.251090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.11.18.67:42201)
0924 19:17:36.251493 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:36.251501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:36.251523 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:17:36.251593 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:36.251598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:36.251676 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:36.251781 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:36.253797 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:36.255101 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:36.256863 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:36.257309 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:36.257493 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:36.262742 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:36.262759 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:36.262775 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:36.262862 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:36.266174 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:36.267021 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:36.267034 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:36.267036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:36.267218 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:36.267688 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:36.267697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:36.267700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:36.268158 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:36.268406 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:36.268930 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 19:17:36.269185 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 19:17:36.270401 30073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:36.367928 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 29950
I20260924 19:17:36.386070 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 29589
I20260924 19:17:36.387277 29651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:36.503688 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 29589
2026-09-24T19:17:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4579 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T19:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:36Z Disabled control of system clock
I20260924 19:17:36.584959 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36523
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:36523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:17:36.929435 30132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:36.929754 30132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:36.929838 30132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:36.937304 30132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:36.937459 30132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:36.937500 30132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:36.937549 30132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:36.937598 30132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:36.947094 30132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:36523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:36.948369 30132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:36.950259 30132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:36.962704 30137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:36.962713 30138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:36.963393 30140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:36.963551 30132 server_base.cc:1034] running on GCE node
I20260924 19:17:36.964466 30132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:36.966215 30132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:36.967485 30132 hybrid_clock.cc:648] HybridClock initialized: now 1790277456967426 us; error 83 us; skew 500 ppm
I20260924 19:17:36.967963 30132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:36.970642 30132 webserver.cc:460] Webserver started at http://127.11.18.126:38783/ using document root <none> and password file <none>
I20260924 19:17:36.971424 30132 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:36.971571 30132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:36.972057 30132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:36.975394 30132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "6ec446522e0342f5aaebe3bdc39a479c"
format_stamp: "Formatted at 2026-09-24 19:17:36 on dist-test-slave-0lgr"
server_key: "f8df63000b58e25d6f416de07e31a12b"
server_key_iv: "5d97822f638ed78f1e12f6e69e70d183"
server_key_version: "encryptionkey@0"
I20260924 19:17:36.976337 30132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "6ec446522e0342f5aaebe3bdc39a479c"
format_stamp: "Formatted at 2026-09-24 19:17:36 on dist-test-slave-0lgr"
server_key: "f8df63000b58e25d6f416de07e31a12b"
server_key_iv: "5d97822f638ed78f1e12f6e69e70d183"
server_key_version: "encryptionkey@0"
I20260924 19:17:36.982544 30132 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 19:17:36.986696 30146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:36.988595 30132 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260924 19:17:36.988883 30132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "6ec446522e0342f5aaebe3bdc39a479c"
format_stamp: "Formatted at 2026-09-24 19:17:36 on dist-test-slave-0lgr"
server_key: "f8df63000b58e25d6f416de07e31a12b"
server_key_iv: "5d97822f638ed78f1e12f6e69e70d183"
server_key_version: "encryptionkey@0"
I20260924 19:17:36.989156 30132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:37.022902 30132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:37.023907 30132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:37.024258 30132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:37.044946 30132 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:36523
I20260924 19:17:37.045003 30197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:36523 every 8 connection(s)
I20260924 19:17:37.046502 30132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:37.051026 30198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:37.051494 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 30132
I20260924 19:17:37.051681 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:37.052170 11337 external_mini_cluster.cc:1428] Setting key d2f5492a2172c877456b47ca541b8b01
I20260924 19:17:37.067955 30198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c: Bootstrap starting.
I20260924 19:17:37.070504 30201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:37.054261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.11.18.126:36523)
0924 19:17:37.055513 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:37.055529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:37.055585 (+ 56us) server_negotiation.cc:407] Connection header received
0924 19:17:37.058729 (+ 3144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:37.058764 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:37.059535 (+ 771us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:37.060088 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:37.060682 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:37.062779 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:37.064392 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:37.065662 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:37.065847 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:37.067288 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:37.067303 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:37.067323 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:37.068115 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:37.068225 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:37.068254 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:37.068386 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:37.068628 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 19:17:37.068984 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":362,"threads_started":1}
I20260924 19:17:37.072566 30198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:37.073977 30198 log.cc:824] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:37.077252 30198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c: No bootstrap required, opened a new log
I20260924 19:17:37.086326 30198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec446522e0342f5aaebe3bdc39a479c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36523 } }
I20260924 19:17:37.086887 30198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:37.087018 30198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec446522e0342f5aaebe3bdc39a479c, State: Initialized, Role: FOLLOWER
I20260924 19:17:37.087693 30198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec446522e0342f5aaebe3bdc39a479c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36523 } }
I20260924 19:17:37.087975 30198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:37.088136 30198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:37.088327 30198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:37.090859 30198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec446522e0342f5aaebe3bdc39a479c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36523 } }
I20260924 19:17:37.091445 30198 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ec446522e0342f5aaebe3bdc39a479c; no voters:
I20260924 19:17:37.092175 30198 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:37.092347 30203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:37.093590 30203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [term 1 LEADER]: Becoming Leader. State: Replica: 6ec446522e0342f5aaebe3bdc39a479c, State: Running, Role: LEADER
I20260924 19:17:37.094286 30203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec446522e0342f5aaebe3bdc39a479c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36523 } }
I20260924 19:17:37.094820 30198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:37.097419 30204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ec446522e0342f5aaebe3bdc39a479c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec446522e0342f5aaebe3bdc39a479c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36523 } } }
I20260924 19:17:37.097975 30204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:37.097790 30205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec446522e0342f5aaebe3bdc39a479c. Latest consensus state: current_term: 1 leader_uuid: "6ec446522e0342f5aaebe3bdc39a479c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec446522e0342f5aaebe3bdc39a479c" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36523 } } }
I20260924 19:17:37.098424 30205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:37.099740 30212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:37.105902 30212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:37.115849 30212 catalog_manager.cc:1347] Generated new cluster ID: cc496958f8b8496ea8773af322428abf
I20260924 19:17:37.116022 30212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:37.126895 30212 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:37.128511 30212 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:37.137734 30212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c: Generated new TSK 0
I20260924 19:17:37.138715 30212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:37.149484 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36523
--builtin_ntp_servers=127.11.18.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:17:37.523409 30222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:37.523742 30222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:37.523841 30222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:37.532064 30222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:37.532186 30222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:37.532295 30222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:17:37.541704 30222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:37.543004 30222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:37.544713 30222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:37.559582 30228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:37.559598 30227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:37.559930 30230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:37.560494 30222 server_base.cc:1034] running on GCE node
I20260924 19:17:37.561197 30222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:37.562289 30222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:37.563586 30222 hybrid_clock.cc:648] HybridClock initialized: now 1790277457563501 us; error 64 us; skew 500 ppm
I20260924 19:17:37.563998 30222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:37.566811 30222 webserver.cc:460] Webserver started at http://127.11.18.65:42417/ using document root <none> and password file <none>
I20260924 19:17:37.567595 30222 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:37.567737 30222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:37.568089 30222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:37.571229 30222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "14a7f94e5d2947d0ad6432dff39c973a"
format_stamp: "Formatted at 2026-09-24 19:17:37 on dist-test-slave-0lgr"
server_key: "4c09ac3f9298b7b1e925ecfb2d7e4f67"
server_key_iv: "e3cd929068dea2029115ef2fe3542c68"
server_key_version: "encryptionkey@0"
I20260924 19:17:37.572184 30222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "14a7f94e5d2947d0ad6432dff39c973a"
format_stamp: "Formatted at 2026-09-24 19:17:37 on dist-test-slave-0lgr"
server_key: "4c09ac3f9298b7b1e925ecfb2d7e4f67"
server_key_iv: "e3cd929068dea2029115ef2fe3542c68"
server_key_version: "encryptionkey@0"
I20260924 19:17:37.578055 30222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 19:17:37.582472 30236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:37.584074 30222 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:17:37.584302 30222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "14a7f94e5d2947d0ad6432dff39c973a"
format_stamp: "Formatted at 2026-09-24 19:17:37 on dist-test-slave-0lgr"
server_key: "4c09ac3f9298b7b1e925ecfb2d7e4f67"
server_key_iv: "e3cd929068dea2029115ef2fe3542c68"
server_key_version: "encryptionkey@0"
I20260924 19:17:37.584554 30222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:37.615626 30222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:37.616575 30222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:37.616916 30222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:37.618081 30222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:37.619582 30222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:37.619673 30222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:37.619813 30222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:37.619903 30222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:37.656243 30222 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:39175
I20260924 19:17:37.656361 30348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:39175 every 8 connection(s)
I20260924 19:17:37.657853 30222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:17:37.658143 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 30222
I20260924 19:17:37.658316 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:17:37.659046 11337 external_mini_cluster.cc:1428] Setting key 66238615b8b29d9bc30fc6d10754654d
I20260924 19:17:37.662736 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36523
--builtin_ntp_servers=127.11.18.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:17:37.677533 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:37.660816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:34839 (local address 127.11.18.126:36523)
0924 19:17:37.666058 (+ 5242us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:37.666066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:37.666999 (+ 933us) server_negotiation.cc:407] Connection header received
0924 19:17:37.670013 (+ 3014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:37.670018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:37.670095 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:37.670197 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:37.671874 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:37.672656 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:37.674472 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:37.674899 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:37.675041 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:37.676559 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:37.676565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:37.676568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:37.676589 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:37.676632 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:37.676644 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:37.676744 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:37.677017 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 19:17:37.677130 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5068,"thread_start_us":4910,"threads_started":1}
I20260924 19:17:37.678611 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:37.661437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36523 (local address 127.11.18.65:34839)
0924 19:17:37.666769 (+ 5332us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:37.666833 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:37.669634 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:37.670513 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:37.670528 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:37.670661 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:37.671543 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:37.671574 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:37.672870 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:37.672874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:37.674306 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:37.674320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:37.675157 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:37.675165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:37.675416 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:37.676174 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:37.676177 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:37.676224 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:37.676344 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:37.676743 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:37.676757 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:37.676777 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 19:17:37.677009 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 19:17:37.677476 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4683,"thread_start_us":4481,"threads_started":1}
I20260924 19:17:37.680335 30349 heartbeater.cc:347] Connected to a master server at 127.11.18.126:36523
I20260924 19:17:37.680706 30349 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:37.681716 30349 heartbeater.cc:511] Master 127.11.18.126:36523 requested a full tablet report, sending...
I20260924 19:17:37.684201 30163 ts_manager.cc:194] Registered new tserver with Master: 14a7f94e5d2947d0ad6432dff39c973a (127.11.18.65:39175)
I20260924 19:17:37.685941 30163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.65:34839
W20260924 19:17:38.006307 30353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:38.006669 30353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:38.006824 30353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:38.015053 30353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:38.015213 30353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:38.015416 30353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:17:38.024899 30353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:38.026345 30353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:38.028132 30353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:38.042364 30358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:38.042384 30361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:38.043236 30359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:38.043524 30353 server_base.cc:1034] running on GCE node
I20260924 19:17:38.044148 30353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:38.045229 30353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:38.046525 30353 hybrid_clock.cc:648] HybridClock initialized: now 1790277458046443 us; error 51 us; skew 500 ppm
I20260924 19:17:38.046972 30353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:38.049731 30353 webserver.cc:460] Webserver started at http://127.11.18.66:34233/ using document root <none> and password file <none>
I20260924 19:17:38.050523 30353 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:38.050668 30353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:38.051010 30353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:38.054297 30353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "8b3c4e25dc744d609efc6252c63ed0e6"
format_stamp: "Formatted at 2026-09-24 19:17:38 on dist-test-slave-0lgr"
server_key: "fe59f96094f9d47a16754cdab989c387"
server_key_iv: "c2ebb769374876d2627949a8e4611e97"
server_key_version: "encryptionkey@0"
I20260924 19:17:38.055182 30353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "8b3c4e25dc744d609efc6252c63ed0e6"
format_stamp: "Formatted at 2026-09-24 19:17:38 on dist-test-slave-0lgr"
server_key: "fe59f96094f9d47a16754cdab989c387"
server_key_iv: "c2ebb769374876d2627949a8e4611e97"
server_key_version: "encryptionkey@0"
I20260924 19:17:38.061187 30353 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 19:17:38.065059 30367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:38.066638 30353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:38.066913 30353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "8b3c4e25dc744d609efc6252c63ed0e6"
format_stamp: "Formatted at 2026-09-24 19:17:38 on dist-test-slave-0lgr"
server_key: "fe59f96094f9d47a16754cdab989c387"
server_key_iv: "c2ebb769374876d2627949a8e4611e97"
server_key_version: "encryptionkey@0"
I20260924 19:17:38.067157 30353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:38.086274 30353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:38.087188 30353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:38.087512 30353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:38.088722 30353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:38.090555 30353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:38.090694 30353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:38.090837 30353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:38.090924 30353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:38.126837 30353 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:35959
I20260924 19:17:38.126859 30479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:35959 every 8 connection(s)
I20260924 19:17:38.128196 30353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:17:38.133178 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 30353
I20260924 19:17:38.133409 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:17:38.133898 11337 external_mini_cluster.cc:1428] Setting key d473d34abed3fe503c5f66f093a3e9ad
I20260924 19:17:38.137035 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36523
--builtin_ntp_servers=127.11.18.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:17:38.141674 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.131133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:53239 (local address 127.11.18.126:36523)
0924 19:17:38.131331 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.131336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.132767 (+ 1431us) server_negotiation.cc:407] Connection header received
0924 19:17:38.134914 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.134920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.135012 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.135146 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:38.136392 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.137149 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.138850 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.139263 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.139405 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.140821 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:38.140826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:38.140829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:38.140850 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:38.140881 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:38.140893 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:38.140990 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.141224 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.141354 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:17:38.142722 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.131645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36523 (local address 127.11.18.66:53239)
0924 19:17:38.132587 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:38.132624 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:38.134602 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:38.135382 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:38.135390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:38.135480 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:38.136216 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.136236 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.137403 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.137409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.138687 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.138699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.139497 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.139504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.139741 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.140474 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:38.140476 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:38.140519 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:38.140600 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:38.140991 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:38.141004 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:38.141014 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:17:38.141202 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 19:17:38.141665 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":320,"threads_started":1}
I20260924 19:17:38.144351 30480 heartbeater.cc:347] Connected to a master server at 127.11.18.126:36523
I20260924 19:17:38.144675 30480 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:38.145542 30480 heartbeater.cc:511] Master 127.11.18.126:36523 requested a full tablet report, sending...
I20260924 19:17:38.147213 30163 ts_manager.cc:194] Registered new tserver with Master: 8b3c4e25dc744d609efc6252c63ed0e6 (127.11.18.66:35959)
I20260924 19:17:38.148133 30163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.66:53239
W20260924 19:17:38.483803 30483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:38.484099 30483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:38.484242 30483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:38.491678 30483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:38.491787 30483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:38.491959 30483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:17:38.501686 30483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:38.503010 30483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:38.504779 30483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:38.518929 30488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:38.518817 30489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:38.518898 30491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:38.519778 30483 server_base.cc:1034] running on GCE node
I20260924 19:17:38.520514 30483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:38.521664 30483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:38.522930 30483 hybrid_clock.cc:648] HybridClock initialized: now 1790277458522889 us; error 36 us; skew 500 ppm
I20260924 19:17:38.523329 30483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:38.526369 30483 webserver.cc:460] Webserver started at http://127.11.18.67:35063/ using document root <none> and password file <none>
I20260924 19:17:38.527277 30483 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:38.527418 30483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:38.527762 30483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:38.530826 30483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "7d4a0505b87045a5bc37db970fbbd727"
format_stamp: "Formatted at 2026-09-24 19:17:38 on dist-test-slave-0lgr"
server_key: "6cf602da9770c8e81a1d243fbee9e4e8"
server_key_iv: "f8ca149c6f178cb4bda2dbde1214e47f"
server_key_version: "encryptionkey@0"
I20260924 19:17:38.531682 30483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "7d4a0505b87045a5bc37db970fbbd727"
format_stamp: "Formatted at 2026-09-24 19:17:38 on dist-test-slave-0lgr"
server_key: "6cf602da9770c8e81a1d243fbee9e4e8"
server_key_iv: "f8ca149c6f178cb4bda2dbde1214e47f"
server_key_version: "encryptionkey@0"
I20260924 19:17:38.537146 30483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:17:38.540963 30497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:38.542608 30483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:38.542840 30483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "7d4a0505b87045a5bc37db970fbbd727"
format_stamp: "Formatted at 2026-09-24 19:17:38 on dist-test-slave-0lgr"
server_key: "6cf602da9770c8e81a1d243fbee9e4e8"
server_key_iv: "f8ca149c6f178cb4bda2dbde1214e47f"
server_key_version: "encryptionkey@0"
I20260924 19:17:38.543107 30483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:38.563288 30483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:38.564247 30483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:38.564589 30483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:38.565699 30483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:38.567312 30483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:38.567449 30483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:38.567600 30483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:38.567698 30483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:38.610551 30483 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:33771
I20260924 19:17:38.610598 30609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:33771 every 8 connection(s)
I20260924 19:17:38.611991 30483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:17:38.615662 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 30483
I20260924 19:17:38.615936 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:17:38.616556 11337 external_mini_cluster.cc:1428] Setting key 46dc28f0bd5ae2c230370e1594c3cec2
I20260924 19:17:38.626283 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.614750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:50839 (local address 127.11.18.126:36523)
0924 19:17:38.614996 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.615004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.616912 (+ 1908us) server_negotiation.cc:407] Connection header received
0924 19:17:38.619268 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.619274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.619425 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.619546 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:38.620659 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.621554 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.623228 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.623660 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.623816 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.625368 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:38.625377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:38.625382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:38.625408 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:38.625451 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:38.625465 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:38.625570 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.625807 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.625926 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 19:17:38.627362 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.615493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:36523 (local address 127.11.18.67:50839)
0924 19:17:38.616706 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:38.616765 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:38.618996 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:38.619789 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:38.619797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:38.619893 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:38.620486 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.620504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.621705 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.621718 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.623073 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.623086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.623908 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.623918 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.624176 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.624979 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:38.624982 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:38.625024 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:38.625104 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:38.625568 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:38.625581 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:38.625592 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:17:38.625797 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 19:17:38.626222 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":432,"threads_started":1}
I20260924 19:17:38.628888 30610 heartbeater.cc:347] Connected to a master server at 127.11.18.126:36523
I20260924 19:17:38.629280 30610 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:38.630265 30610 heartbeater.cc:511] Master 127.11.18.126:36523 requested a full tablet report, sending...
I20260924 19:17:38.631754 30163 ts_manager.cc:194] Registered new tserver with Master: 7d4a0505b87045a5bc37db970fbbd727 (127.11.18.67:33771)
I20260924 19:17:38.632433 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:17:38.632617 30163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.67:50839
I20260924 19:17:38.643553 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.637170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.11.18.126:36523)
0924 19:17:38.637523 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.637530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.637828 (+ 298us) server_negotiation.cc:407] Connection header received
0924 19:17:38.638098 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.638102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.638165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.638270 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:38.638777 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.639497 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.640889 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.641379 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.641539 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.642750 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:38.642758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:38.642767 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:38.642786 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:38.642818 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:38.642829 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:38.642985 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.643196 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.643259 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:17:38.648583 30163 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:17:38.655628 30163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:17:38.689303 30349 heartbeater.cc:503] Master 127.11.18.126:36523 was elected leader, sending a full tablet report...
I20260924 19:17:38.697010 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.670137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51420)
0924 19:17:38.678718 (+ 8581us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:38.678786 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:38.679060 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:38.688879 (+ 9819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:38.688892 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:38.688948 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:38.689434 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.689448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.691332 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.691341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.693876 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.693897 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.695926 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.695932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.696149 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.696214 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 19:17:38.696341 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 19:17:38.696562 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8217,"mutex_wait_us":84,"thread_start_us":18398,"threads_started":1}
I20260924 19:17:38.697927 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.670912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.11.18.65:39175)
0924 19:17:38.687955 (+ 17043us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.687969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.688012 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:17:38.688156 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.688168 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.688520 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.689396 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:38.689637 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.691166 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.694121 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.695672 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.695875 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.696190 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.697348 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.697563 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16794,"thread_start_us":16638,"threads_started":1}
I20260924 19:17:38.698297 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.670147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:33771 (local address 127.0.0.1:33294)
0924 19:17:38.686820 (+ 16673us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:38.686859 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:38.687044 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:38.690680 (+ 3636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:38.690687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:38.690717 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:38.691120 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.691136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.693286 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.693292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.695498 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.695513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.697490 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.697504 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.697669 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.697699 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:17:38.697776 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:17:38.697917 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16469,"mutex_wait_us":3,"thread_start_us":16982,"threads_started":1}
I20260924 19:17:38.698598 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.673404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.11.18.66:35959)
0924 19:17:38.681675 (+ 8271us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.681701 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.688969 (+ 7268us) server_negotiation.cc:407] Connection header received
0924 19:17:38.689399 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.689409 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.689702 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.689910 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:38.690813 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.692421 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.695133 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.696689 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.696834 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.697085 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.697669 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.697845 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8074,"thread_start_us":7926,"threads_started":1}
I20260924 19:17:38.698899 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.673370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:35959 (local address 127.0.0.1:33892)
0924 19:17:38.688825 (+ 15455us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:38.688849 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:38.689103 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:38.690154 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:38.690159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:38.690191 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:38.690622 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.690638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.692616 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.692622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.694928 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.694947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.697229 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.697236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.697435 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.697479 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:17:38.698397 (+ 918us) client_negotiation.cc:240] Negotiation successful
0924 19:17:38.698552 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15307,"mutex_wait_us":26,"thread_start_us":4791,"threads_started":1}
I20260924 19:17:38.699471 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.670255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.11.18.67:33771)
0924 19:17:38.690096 (+ 19841us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.690105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.690137 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:17:38.690223 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.690232 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.690465 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.691325 (+ 860us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:38.691591 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.693097 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.695699 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.697036 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.697166 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.698886 (+ 1720us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.698955 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.699108 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19650,"thread_start_us":18754,"threads_started":1}
I20260924 19:17:38.702270 30284 tablet_service.cc:1467] Processing CreateTablet for tablet a9554b28c4c3431080d635d7accb9806 (DEFAULT_TABLE table=test-table [id=040d5e475e10496898fecae81070741c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:38.702696 30415 tablet_service.cc:1467] Processing CreateTablet for tablet a9554b28c4c3431080d635d7accb9806 (DEFAULT_TABLE table=test-table [id=040d5e475e10496898fecae81070741c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:38.702769 30545 tablet_service.cc:1467] Processing CreateTablet for tablet a9554b28c4c3431080d635d7accb9806 (DEFAULT_TABLE table=test-table [id=040d5e475e10496898fecae81070741c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:38.704069 30284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9554b28c4c3431080d635d7accb9806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:38.704344 30415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9554b28c4c3431080d635d7accb9806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:38.704339 30545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9554b28c4c3431080d635d7accb9806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:38.717716 30628 tablet_bootstrap.cc:492] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a: Bootstrap starting.
I20260924 19:17:38.720489 30630 tablet_bootstrap.cc:492] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Bootstrap starting.
I20260924 19:17:38.721302 30629 tablet_bootstrap.cc:492] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Bootstrap starting.
I20260924 19:17:38.721735 30628 tablet_bootstrap.cc:654] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:38.723117 30628 log.cc:824] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:38.725171 30630 tablet_bootstrap.cc:654] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:38.726238 30629 tablet_bootstrap.cc:654] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:38.726490 30630 log.cc:824] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:38.726751 30628 tablet_bootstrap.cc:492] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a: No bootstrap required, opened a new log
I20260924 19:17:38.727268 30628 ts_tablet_manager.cc:1397] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 19:17:38.727484 30629 log.cc:824] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:38.732197 30630 tablet_bootstrap.cc:492] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: No bootstrap required, opened a new log
I20260924 19:17:38.732197 30629 tablet_bootstrap.cc:492] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: No bootstrap required, opened a new log
I20260924 19:17:38.732496 30630 ts_tablet_manager.cc:1397] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 19:17:38.732494 30629 ts_tablet_manager.cc:1397] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260924 19:17:38.736145 30628 raft_consensus.cc:348] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.736544 30628 raft_consensus.cc:374] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:38.736680 30628 raft_consensus.cc:729] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a7f94e5d2947d0ad6432dff39c973a, State: Initialized, Role: FOLLOWER
I20260924 19:17:38.737418 30628 consensus_queue.cc:260] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.739298 30628 ts_tablet_manager.cc:1428] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 19:17:38.743764 30630 raft_consensus.cc:348] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.744304 30630 raft_consensus.cc:374] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:38.744462 30630 raft_consensus.cc:729] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3c4e25dc744d609efc6252c63ed0e6, State: Initialized, Role: FOLLOWER
I20260924 19:17:38.744202 30629 raft_consensus.cc:348] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.744635 30629 raft_consensus.cc:374] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:38.744735 30629 raft_consensus.cc:729] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4a0505b87045a5bc37db970fbbd727, State: Initialized, Role: FOLLOWER
I20260924 19:17:38.745172 30630 consensus_queue.cc:260] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.745309 30629 consensus_queue.cc:260] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.746568 30480 heartbeater.cc:503] Master 127.11.18.126:36523 was elected leader, sending a full tablet report...
I20260924 19:17:38.747080 30610 heartbeater.cc:503] Master 127.11.18.126:36523 was elected leader, sending a full tablet report...
I20260924 19:17:38.747429 30630 ts_tablet_manager.cc:1428] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260924 19:17:38.747617 30629 ts_tablet_manager.cc:1428] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260924 19:17:38.809530 30636 raft_consensus.cc:482] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:38.809897 30636 raft_consensus.cc:504] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.811810 30636 leader_election.cc:288] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14a7f94e5d2947d0ad6432dff39c973a (127.11.18.65:39175), 8b3c4e25dc744d609efc6252c63ed0e6 (127.11.18.66:35959)
I20260924 19:17:38.819013 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.812280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:35959 (local address 127.11.18.67:45835)
0924 19:17:38.812508 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:38.812529 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:38.812669 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:38.813251 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:38.813257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:38.813285 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:38.813557 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.813594 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.814900 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.814904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.817082 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.817098 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.818357 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.818361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.818529 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.818556 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 19:17:38.818636 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:17:38.818756 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 19:17:38.819095 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.812478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:45835 (local address 127.11.18.66:35959)
0924 19:17:38.812712 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.812718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.812739 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:17:38.812823 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.812830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.812958 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.813112 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:38.813778 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.814763 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.817267 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.818198 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.818360 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.818432 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.818694 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.818832 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 19:17:38.820153 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.812315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:41533 (local address 127.11.18.65:39175)
0924 19:17:38.812537 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.812543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.813626 (+ 1083us) server_negotiation.cc:407] Connection header received
0924 19:17:38.813877 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.813883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.813991 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.814116 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:38.814806 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.815524 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.818058 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.819278 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.819450 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.819516 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.819781 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.819905 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:17:38.820237 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.812280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.11.18.67:41533)
0924 19:17:38.813473 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:38.813501 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:38.813651 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:38.814234 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:38.814239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:38.814269 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:38.814622 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.814631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.815736 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.815743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.817787 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:38.817799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.819447 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.819453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:38.819626 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.819647 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 19:17:38.819752 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:17:38.819884 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":261,"thread_start_us":422,"threads_started":1}
I20260924 19:17:38.820276 30435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9554b28c4c3431080d635d7accb9806" candidate_uuid: "7d4a0505b87045a5bc37db970fbbd727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" is_pre_election: true
I20260924 19:17:38.820943 30435 raft_consensus.cc:2454] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4a0505b87045a5bc37db970fbbd727 in term 0.
I20260924 19:17:38.821224 30304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9554b28c4c3431080d635d7accb9806" candidate_uuid: "7d4a0505b87045a5bc37db970fbbd727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14a7f94e5d2947d0ad6432dff39c973a" is_pre_election: true
I20260924 19:17:38.821784 30499 leader_election.cc:302] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4a0505b87045a5bc37db970fbbd727, 8b3c4e25dc744d609efc6252c63ed0e6; no voters:
I20260924 19:17:38.821841 30304 raft_consensus.cc:2454] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4a0505b87045a5bc37db970fbbd727 in term 0.
I20260924 19:17:38.822279 30636 raft_consensus.cc:2781] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:17:38.822458 30636 raft_consensus.cc:482] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:17:38.822608 30636 raft_consensus.cc:3037] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:38.824889 30636 raft_consensus.cc:504] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.825661 30636 leader_election.cc:288] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [CANDIDATE]: Term 1 election: Requested vote from peers 14a7f94e5d2947d0ad6432dff39c973a (127.11.18.65:39175), 8b3c4e25dc744d609efc6252c63ed0e6 (127.11.18.66:35959)
I20260924 19:17:38.826193 30435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9554b28c4c3431080d635d7accb9806" candidate_uuid: "7d4a0505b87045a5bc37db970fbbd727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3c4e25dc744d609efc6252c63ed0e6"
I20260924 19:17:38.826278 30304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9554b28c4c3431080d635d7accb9806" candidate_uuid: "7d4a0505b87045a5bc37db970fbbd727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14a7f94e5d2947d0ad6432dff39c973a"
I20260924 19:17:38.826417 30435 raft_consensus.cc:3037] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:38.826521 30304 raft_consensus.cc:3037] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:38.829500 30304 raft_consensus.cc:2454] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4a0505b87045a5bc37db970fbbd727 in term 1.
I20260924 19:17:38.829579 30435 raft_consensus.cc:2454] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4a0505b87045a5bc37db970fbbd727 in term 1.
I20260924 19:17:38.830099 30499 leader_election.cc:302] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14a7f94e5d2947d0ad6432dff39c973a, 7d4a0505b87045a5bc37db970fbbd727, 8b3c4e25dc744d609efc6252c63ed0e6; no voters:
I20260924 19:17:38.830524 30636 raft_consensus.cc:2781] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:38.831017 30636 raft_consensus.cc:686] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4a0505b87045a5bc37db970fbbd727, State: Running, Role: LEADER
I20260924 19:17:38.831571 30636 consensus_queue.cc:237] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } }
I20260924 19:17:38.836213 30162 catalog_manager.cc:5495] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d4a0505b87045a5bc37db970fbbd727 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "7d4a0505b87045a5bc37db970fbbd727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4a0505b87045a5bc37db970fbbd727" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 33771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 } health_report { overall_health: UNKNOWN } } }
I20260924 19:17:38.858630 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.852217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.11.18.67:33771)
0924 19:17:38.852419 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.852424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.852575 (+ 151us) server_negotiation.cc:407] Connection header received
0924 19:17:38.852760 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.852765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.852873 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.853007 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:17:38.853479 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.854304 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.855880 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.856303 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.856423 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.857084 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:17:38.857822 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:17:38.857983 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.858389 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.858431 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 19:17:38.863816 30611 tablet.cc:2369] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:38.867451 30435 raft_consensus.cc:1261] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 1 FOLLOWER]: Refusing update from remote peer 7d4a0505b87045a5bc37db970fbbd727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:38.867465 30304 raft_consensus.cc:1261] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 1 FOLLOWER]: Refusing update from remote peer 7d4a0505b87045a5bc37db970fbbd727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:38.868553 30638 consensus_queue.cc:1035] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b3c4e25dc744d609efc6252c63ed0e6" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 35959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:38.868933 30636 consensus_queue.cc:1035] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a7f94e5d2947d0ad6432dff39c973a" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:17:38.880596 30481 tablet.cc:2369] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:38.882613 30641 mvcc.cc:205] Tried to move back new op lower bound from 7332976471510794240 to 7332976471378255872. Current Snapshot: MvccSnapshot[applied={T|T < 7332976471510794240}]
I20260924 19:17:38.909993 30162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:38.910377 30162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:38.914750 30162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ec446522e0342f5aaebe3bdc39a479c: Sending DeleteTablet for 3 replicas of tablet a9554b28c4c3431080d635d7accb9806
I20260924 19:17:38.916113 30284 tablet_service.cc:1514] Processing DeleteTablet for tablet a9554b28c4c3431080d635d7accb9806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:38 UTC) from {username='slave'} at 127.0.0.1:51420
I20260924 19:17:38.916172 30545 tablet_service.cc:1514] Processing DeleteTablet for tablet a9554b28c4c3431080d635d7accb9806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:38 UTC) from {username='slave'} at 127.0.0.1:33294
I20260924 19:17:38.916368 30415 tablet_service.cc:1514] Processing DeleteTablet for tablet a9554b28c4c3431080d635d7accb9806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:38 UTC) from {username='slave'} at 127.0.0.1:33892
I20260924 19:17:38.918762 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 30222
I20260924 19:17:38.935606 30651 tablet_replica.cc:330] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a: stopping tablet replica
I20260924 19:17:38.935760 30653 tablet_replica.cc:330] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: stopping tablet replica
I20260924 19:17:38.936311 30653 raft_consensus.cc:2229] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:38.936930 30653 raft_consensus.cc:2258] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:38.937878 30652 tablet_replica.cc:330] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: stopping tablet replica
I20260924 19:17:38.938187 30651 raft_consensus.cc:2229] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:38.939132 30651 raft_consensus.cc:2258] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:38.939874 30652 raft_consensus.cc:2229] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:17:38.940040 30653 ts_tablet_manager.cc:1905] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:38.940537 30652 raft_consensus.cc:2258] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:38.942425 30651 ts_tablet_manager.cc:1905] T a9554b28c4c3431080d635d7accb9806 P 14a7f94e5d2947d0ad6432dff39c973a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:38.945763 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:38.920900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.11.18.65:39175)
0924 19:17:38.921169 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:38.921179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:38.925063 (+ 3884us) server_negotiation.cc:407] Connection header received
0924 19:17:38.926021 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:38.926028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:38.926185 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:38.926277 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:38.927656 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.929380 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.936191 (+ 6811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:38.937284 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:38.937495 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:38.943714 (+ 6219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:38.943728 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:38.943745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:38.944583 (+ 838us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:38.944652 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:38.944680 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:38.944823 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:38.945136 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 19:17:38.945281 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 19:17:38.947424 30653 ts_tablet_manager.cc:1918] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:38.947686 30653 log.cc:1196] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/a9554b28c4c3431080d635d7accb9806
I20260924 19:17:38.947705 30344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:38.948338 30653 ts_tablet_manager.cc:1939] T a9554b28c4c3431080d635d7accb9806 P 8b3c4e25dc744d609efc6252c63ed0e6: Deleting consensus metadata
I20260924 19:17:38.950202 30652 ts_tablet_manager.cc:1905] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:38.950490 30148 catalog_manager.cc:4841] TS 8b3c4e25dc744d609efc6252c63ed0e6 (127.11.18.66:35959): tablet a9554b28c4c3431080d635d7accb9806 (table test-table [id=040d5e475e10496898fecae81070741c]) successfully deleted
I20260924 19:17:38.958271 30652 ts_tablet_manager.cc:1918] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:38.958484 30652 log.cc:1196] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/a9554b28c4c3431080d635d7accb9806
I20260924 19:17:38.959034 30652 ts_tablet_manager.cc:1939] T a9554b28c4c3431080d635d7accb9806 P 7d4a0505b87045a5bc37db970fbbd727: Deleting consensus metadata
I20260924 19:17:38.960654 30149 catalog_manager.cc:4841] TS 7d4a0505b87045a5bc37db970fbbd727 (127.11.18.67:33771): tablet a9554b28c4c3431080d635d7accb9806 (table test-table [id=040d5e475e10496898fecae81070741c]) successfully deleted
I20260924 19:17:39.066468 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 30222
W20260924 19:17:39.085531 30147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.18.65:39175 (error 108)
I20260924 19:17:39.085700 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 30353
I20260924 19:17:39.087779 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.087223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51436)
0924 19:17:39.087481 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:39.087596 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260924 19:17:39.088045 30147 catalog_manager.cc:4584] TS 14a7f94e5d2947d0ad6432dff39c973a (127.11.18.65:39175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9554b28c4c3431080d635d7accb9806: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
I20260924 19:17:39.093205 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.086756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.11.18.66:35959)
0924 19:17:39.086940 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:39.086944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:39.086965 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:17:39.087022 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:39.087025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:39.087106 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:39.087237 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:39.087637 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:39.088665 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:39.090113 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:39.090537 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:39.090677 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:39.091806 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:39.091815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:39.091829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:39.092411 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:39.092460 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:39.092480 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:39.092611 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:39.092751 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 19:17:39.092873 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:17:39.093950 30475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:39.142140 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.141562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51438)
0924 19:17:39.141819 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:39.141937 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 19:17:39.143381 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.142949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51450)
0924 19:17:39.143122 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:39.143215 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:17:39.187418 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 30353
I20260924 19:17:39.204373 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.203780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51464)
0924 19:17:39.204038 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:39.204153 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 19:17:39.205771 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.205233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51480)
0924 19:17:39.205463 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:39.205566 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 19:17:39.207201 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 30483
I20260924 19:17:39.215379 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.208042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.11.18.67:33771)
0924 19:17:39.208235 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:39.208241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:39.208263 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:17:39.208390 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:39.208396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:39.208498 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:39.208620 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:39.209019 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:39.210212 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:39.211553 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:39.212212 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:39.212369 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:39.213563 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:39.213574 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:39.213594 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:39.214459 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:39.214525 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:39.214552 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:39.214681 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:39.214856 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:17:39.214965 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:17:39.216236 30605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:39.286501 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.285966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51490)
0924 19:17:39.286206 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:39.286310 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 19:17:39.287915 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.287491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51500)
0924 19:17:39.287711 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:39.287784 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 19:17:39.344291 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 30483
I20260924 19:17:39.363528 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 30132
I20260924 19:17:39.364925 30193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:39.478214 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:39.477531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39175 (local address 127.0.0.1:51502)
0924 19:17:39.477771 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:39.477972 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:17:39.478343 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 30132
2026-09-24T19:17:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:39 dist-test-slave-0lgr krb5kdc[30662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:39 dist-test-slave-0lgr krb5kdc[30662](info): set up 2 sockets
Sep 24 19:17:39 dist-test-slave-0lgr krb5kdc[30662](info): commencing operation
krb5kdc: starting...
W20260924 19:17:40.673883 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:40 dist-test-slave-0lgr krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:17:40.942085 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41101
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:41.288089 30675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:41.288411 30675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:41.288514 30675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:41.296998 30675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:41.297125 30675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:41.297173 30675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:41.297223 30675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:41.297269 30675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:41.307648 30675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:41.309002 30675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:41.310292 30675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:41.322790 30683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:41.322974 30680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:41.322916 30681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:41.324299 30675 server_base.cc:1034] running on GCE node
I20260924 19:17:41.325618 30675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:41.327247 30675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:41.328531 30675 hybrid_clock.cc:648] HybridClock initialized: now 1790277461328468 us; error 56 us; skew 500 ppm
I20260924 19:17:41.328946 30675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:41 dist-test-slave-0lgr krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277461, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:41.332608 30675 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:17:41.334594 30675 webserver.cc:460] Webserver started at http://127.11.18.126:42295/ using document root <none> and password file <none>
I20260924 19:17:41.335423 30675 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:41.335595 30675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:41.335944 30675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:41.339486 30675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "9f97368e726b4fb6a68ae628beae2220"
format_stamp: "Formatted at 2026-09-24 19:17:41 on dist-test-slave-0lgr"
server_key: "bd3e4db75e07bfcca61c9d38d39e15a5"
server_key_iv: "e1f1f74d1b05aafe8910affeddfb4642"
server_key_version: "encryptionkey@0"
I20260924 19:17:41.340375 30675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "9f97368e726b4fb6a68ae628beae2220"
format_stamp: "Formatted at 2026-09-24 19:17:41 on dist-test-slave-0lgr"
server_key: "bd3e4db75e07bfcca61c9d38d39e15a5"
server_key_iv: "e1f1f74d1b05aafe8910affeddfb4642"
server_key_version: "encryptionkey@0"
I20260924 19:17:41.346364 30675 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:17:41.350396 30690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:41.352075 30675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:17:41.352299 30675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "9f97368e726b4fb6a68ae628beae2220"
format_stamp: "Formatted at 2026-09-24 19:17:41 on dist-test-slave-0lgr"
server_key: "bd3e4db75e07bfcca61c9d38d39e15a5"
server_key_iv: "e1f1f74d1b05aafe8910affeddfb4642"
server_key_version: "encryptionkey@0"
I20260924 19:17:41.352540 30675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:41.374778 30675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:41.379617 30675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:41.379938 30675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:41.400569 30675 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:41101
I20260924 19:17:41.400604 30741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:41101 every 8 connection(s)
I20260924 19:17:41.402148 30675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:41.406450 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:41.408612 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 30675
I20260924 19:17:41.408815 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:41.409255 11337 external_mini_cluster.cc:1428] Setting key 9714679d742d95e68c36b712f9b43f8f
I20260924 19:17:41.419449 30742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220: Bootstrap starting.
I20260924 19:17:41.423588 30742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:41.424880 30742 log.cc:824] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:17:41 dist-test-slave-0lgr krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:41.427914 30742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220: No bootstrap required, opened a new log
I20260924 19:17:41.434866 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:41.411381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.11.18.126:41101)
0924 19:17:41.412248 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:41.412264 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:41.412306 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:17:41.414058 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:41.414082 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:41.414487 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:41.414811 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:41.416659 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:41.418253 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:41.420203 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:41.420907 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:41.421092 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:41.426440 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:41.426450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:41.426466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:41.426508 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:41.430062 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:41.430916 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:41.430928 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:41.430935 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:41.431126 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:41.431592 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:41.431598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:41.431601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:41.432269 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:41.432642 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:41.433174 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 19:17:41.433714 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":404,"threads_started":1}
I20260924 19:17:41.435989 30742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97368e726b4fb6a68ae628beae2220" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41101 } }
I20260924 19:17:41.436429 30742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:41.436547 30742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f97368e726b4fb6a68ae628beae2220, State: Initialized, Role: FOLLOWER
I20260924 19:17:41.437270 30742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97368e726b4fb6a68ae628beae2220" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41101 } }
I20260924 19:17:41.437582 30742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:41.437722 30742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:41.437899 30742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:41.440150 30742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97368e726b4fb6a68ae628beae2220" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41101 } }
I20260924 19:17:41.440675 30742 leader_election.cc:302] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f97368e726b4fb6a68ae628beae2220; no voters:
I20260924 19:17:41.441270 30742 leader_election.cc:288] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:41.441511 30747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:41.442346 30747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [term 1 LEADER]: Becoming Leader. State: Replica: 9f97368e726b4fb6a68ae628beae2220, State: Running, Role: LEADER
I20260924 19:17:41.443130 30747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97368e726b4fb6a68ae628beae2220" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41101 } }
I20260924 19:17:41.443737 30742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:41.446656 30748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f97368e726b4fb6a68ae628beae2220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97368e726b4fb6a68ae628beae2220" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41101 } } }
I20260924 19:17:41.447070 30748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:41.447645 30749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f97368e726b4fb6a68ae628beae2220. Latest consensus state: current_term: 1 leader_uuid: "9f97368e726b4fb6a68ae628beae2220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97368e726b4fb6a68ae628beae2220" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 41101 } } }
I20260924 19:17:41.448010 30749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:41.448222 30755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:41.455621 30755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:41.466363 30755 catalog_manager.cc:1347] Generated new cluster ID: b17b36d9977a4954b9bfedbcd2661539
I20260924 19:17:41.466511 30755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:41.489362 30755 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:41.491094 30755 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:41.502606 30755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220: Generated new TSK 0
I20260924 19:17:41.503628 30755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
Entry for principal kudu/127.11.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab.
I20260924 19:17:41.575047 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41101
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:41.936224 30768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:41.936563 30768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:41.936662 30768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:41.944376 30768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:41.944505 30768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:41.944685 30768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:17:41.955304 30768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.65.keytab
--principal=kudu/127.11.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:41.956840 30768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:41.958346 30768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:41.973100 30774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:41.973230 30773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:41.973093 30776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:41.974223 30768 server_base.cc:1034] running on GCE node
I20260924 19:17:41.974902 30768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:41.976011 30768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:41.977232 30768 hybrid_clock.cc:648] HybridClock initialized: now 1790277461977169 us; error 51 us; skew 500 ppm
I20260924 19:17:41.977658 30768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:41 dist-test-slave-0lgr krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277461, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:41.980967 30768 init.cc:377] Logged in from keytab as kudu/127.11.18.65@KRBTEST.COM (short username kudu)
I20260924 19:17:41.982695 30768 webserver.cc:460] Webserver started at http://127.11.18.65:39247/ using document root <none> and password file <none>
I20260924 19:17:41.983382 30768 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:41.983489 30768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:41.983763 30768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:41.986922 30768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "bc3bf529199f4420b2ea5e41353dd42c"
format_stamp: "Formatted at 2026-09-24 19:17:41 on dist-test-slave-0lgr"
server_key: "65d5b6b307ec410e0ab73c25ed55788a"
server_key_iv: "5dae4af133929e30c581aace606ca921"
server_key_version: "encryptionkey@0"
I20260924 19:17:41.987692 30768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "bc3bf529199f4420b2ea5e41353dd42c"
format_stamp: "Formatted at 2026-09-24 19:17:41 on dist-test-slave-0lgr"
server_key: "65d5b6b307ec410e0ab73c25ed55788a"
server_key_iv: "5dae4af133929e30c581aace606ca921"
server_key_version: "encryptionkey@0"
I20260924 19:17:41.993494 30768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:17:41.997622 30783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:41.999272 30768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:17:41.999588 30768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "bc3bf529199f4420b2ea5e41353dd42c"
format_stamp: "Formatted at 2026-09-24 19:17:41 on dist-test-slave-0lgr"
server_key: "65d5b6b307ec410e0ab73c25ed55788a"
server_key_iv: "5dae4af133929e30c581aace606ca921"
server_key_version: "encryptionkey@0"
I20260924 19:17:41.999794 30768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:42.026026 30768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:42.030738 30768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:42.031067 30768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:42.032234 30768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:42.033733 30768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:42.033808 30768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:42.033979 30768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:42.034042 30768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:42.070182 30768 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:39351
I20260924 19:17:42.070274 30895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:39351 every 8 connection(s)
I20260924 19:17:42.071583 30768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:17:42.072954 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 30768
I20260924 19:17:42.073148 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:17:42.073796 11337 external_mini_cluster.cc:1428] Setting key 4fff9c992dc66b24209d160fc77f52a0
Sep 24 19:17:42 dist-test-slave-0lgr krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277461, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.65@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:42.095508 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:42.074899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:37143 (local address 127.11.18.126:41101)
0924 19:17:42.075329 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:42.075333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:42.076569 (+ 1236us) server_negotiation.cc:407] Connection header received
0924 19:17:42.078752 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:42.078759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:42.078857 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:42.078987 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:42.080743 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:42.082012 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:42.083810 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:42.084368 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:42.084482 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:42.089190 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:42.089195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:42.089199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:42.089237 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:42.092186 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:42.092991 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:42.092996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:42.093000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:42.093137 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:42.093664 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:42.093670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:42.093675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:42.093966 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:42.094131 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:42.094868 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 19:17:42.095102 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":219,"threads_started":1}
I20260924 19:17:42.096537 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:42.075484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41101 (local address 127.11.18.65:37143)
0924 19:17:42.076384 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:42.076433 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:42.078449 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:42.079292 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:42.079303 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:42.079900 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:42.080542 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:42.080563 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:42.082184 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:42.082188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:42.083602 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:42.083618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:42.084667 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:42.084676 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:42.085022 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:42.085916 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:42.085951 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:42.088871 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:42.092349 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:42.092359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:42.092378 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:42.092847 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:42.093257 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:42.093262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:42.093266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:42.093544 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:42.094109 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:42.094118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:42.094486 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 19:17:42.094763 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 19:17:42.095349 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":375,"threads_started":1}
I20260924 19:17:42.098584 30896 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41101
I20260924 19:17:42.098948 30896 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.18.66@KRBTEST.COM; defaulting to no policy
I20260924 19:17:42.099705 30896 heartbeater.cc:511] Master 127.11.18.126:41101 requested a full tablet report, sending...
I20260924 19:17:42.102046 30707 ts_manager.cc:194] Registered new tserver with Master: bc3bf529199f4420b2ea5e41353dd42c (127.11.18.65:39351)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.66@KRBTEST.COM" created.
I20260924 19:17:42.103916 30707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.65@KRBTEST.COM'} at 127.11.18.65:37143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
Entry for principal kudu/127.11.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab.
I20260924 19:17:42.137936 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41101
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:42.504587 30902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:42.504961 30902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:42.505141 30902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:42.512756 30902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:42.512953 30902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:42.513142 30902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:17:42.523459 30902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.66.keytab
--principal=kudu/127.11.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:42.524910 30902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:42.526331 30902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:42.540874 30910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:42.541052 30908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:42.540906 30907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:42.547477 30902 server_base.cc:1034] running on GCE node
I20260924 19:17:42.548310 30902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:42.549614 30902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:42.550915 30902 hybrid_clock.cc:648] HybridClock initialized: now 1790277462550820 us; error 102 us; skew 500 ppm
I20260924 19:17:42.551318 30902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:42 dist-test-slave-0lgr krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277462, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:42.555189 30902 init.cc:377] Logged in from keytab as kudu/127.11.18.66@KRBTEST.COM (short username kudu)
I20260924 19:17:42.557493 30902 webserver.cc:460] Webserver started at http://127.11.18.66:37763/ using document root <none> and password file <none>
I20260924 19:17:42.558405 30902 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:42.558535 30902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:42.558938 30902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:42.562386 30902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "2c6d10ba02a54913801f0f00409376cc"
format_stamp: "Formatted at 2026-09-24 19:17:42 on dist-test-slave-0lgr"
server_key: "08c8d7f593fefb131d8406d600803b2a"
server_key_iv: "53f172975038f0d7562a7a01e7367dc6"
server_key_version: "encryptionkey@0"
I20260924 19:17:42.563290 30902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "2c6d10ba02a54913801f0f00409376cc"
format_stamp: "Formatted at 2026-09-24 19:17:42 on dist-test-slave-0lgr"
server_key: "08c8d7f593fefb131d8406d600803b2a"
server_key_iv: "53f172975038f0d7562a7a01e7367dc6"
server_key_version: "encryptionkey@0"
I20260924 19:17:42.569286 30902 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260924 19:17:42.573289 30917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:42.575034 30902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:42.575222 30902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "2c6d10ba02a54913801f0f00409376cc"
format_stamp: "Formatted at 2026-09-24 19:17:42 on dist-test-slave-0lgr"
server_key: "08c8d7f593fefb131d8406d600803b2a"
server_key_iv: "53f172975038f0d7562a7a01e7367dc6"
server_key_version: "encryptionkey@0"
I20260924 19:17:42.575448 30902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:42.599568 30902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:42.605242 30902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:42.605616 30902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:42.606662 30902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:42.608242 30902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:42.608325 30902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:42.608421 30902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:42.608470 30902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:42.647665 30902 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:34001
I20260924 19:17:42.647797 31029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:34001 every 8 connection(s)
I20260924 19:17:42.649111 30902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:17:42.656514 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 30902
I20260924 19:17:42.656695 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:17:42.657264 11337 external_mini_cluster.cc:1428] Setting key 22e2fddfb9d4d13937ae2cfc2aaa1100
Sep 24 19:17:42 dist-test-slave-0lgr krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277462, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.66@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:42.673565 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:42.651974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:49937 (local address 127.11.18.126:41101)
0924 19:17:42.652449 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:42.652454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:42.653724 (+ 1270us) server_negotiation.cc:407] Connection header received
0924 19:17:42.655802 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:42.655808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:42.655928 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:42.656063 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:42.658143 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:42.659413 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:42.661218 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:42.661826 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:42.661995 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:42.666364 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:42.666370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:42.666374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:42.666412 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:42.669838 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:42.670812 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:42.670817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:42.670820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:42.670980 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:42.671497 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:42.671508 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:42.671511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:42.671819 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:42.671997 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:42.672807 (+ 810us) server_negotiation.cc:299] Negotiation successful
0924 19:17:42.673073 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":246,"threads_started":1}
I20260924 19:17:42.674702 31033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:42.652566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41101 (local address 127.11.18.66:49937)
0924 19:17:42.653552 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:42.653598 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:42.655515 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:42.656332 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:42.656341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:42.657156 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:42.657932 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:42.657952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:42.659611 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:42.659616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:42.661038 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:42.661052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:42.662100 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:42.662109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:42.662383 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:42.663194 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:42.663226 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:42.666067 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:42.670071 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:42.670081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:42.670098 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:42.670640 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:42.671120 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:42.671128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:42.671137 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:42.671354 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:42.671970 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:42.671980 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:42.672396 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 19:17:42.672699 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 19:17:42.673365 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":291,"threads_started":1}
I20260924 19:17:42.676493 31030 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41101
I20260924 19:17:42.676867 31030 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:42.677721 31030 heartbeater.cc:511] Master 127.11.18.126:41101 requested a full tablet report, sending...
I20260924 19:17:42.679484 30707 ts_manager.cc:194] Registered new tserver with Master: 2c6d10ba02a54913801f0f00409376cc (127.11.18.66:34001)
I20260924 19:17:42.680347 30707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.66@KRBTEST.COM'} at 127.11.18.66:49937
WARNING: no policy specified for kudu/127.11.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
Entry for principal kudu/127.11.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab.
I20260924 19:17:42.722011 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41101
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:43.073921 31036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:43.074268 31036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:43.074448 31036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:43.082489 31036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:43.082623 31036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:43.082804 31036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:17:43.093255 31036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.67.keytab
--principal=kudu/127.11.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:43.094772 31036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:43.096136 31036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:17:43.108062 30896 heartbeater.cc:503] Master 127.11.18.126:41101 was elected leader, sending a full tablet report...
W20260924 19:17:43.111418 31041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:43.111330 31044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:43.111353 31042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:43.113505 31036 server_base.cc:1034] running on GCE node
I20260924 19:17:43.114245 31036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:43.115351 31036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:43.116616 31036 hybrid_clock.cc:648] HybridClock initialized: now 1790277463116563 us; error 40 us; skew 500 ppm
I20260924 19:17:43.117096 31036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:43 dist-test-slave-0lgr krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277463, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:43.120865 31036 init.cc:377] Logged in from keytab as kudu/127.11.18.67@KRBTEST.COM (short username kudu)
I20260924 19:17:43.122686 31036 webserver.cc:460] Webserver started at http://127.11.18.67:46297/ using document root <none> and password file <none>
I20260924 19:17:43.123473 31036 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:43.123641 31036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:43.124004 31036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:43.127432 31036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "4b75bbfad2264396ad61062d808ad194"
format_stamp: "Formatted at 2026-09-24 19:17:43 on dist-test-slave-0lgr"
server_key: "e280a8bdf242704258d3acd1f9583d81"
server_key_iv: "c5df22e8c25933eeebd59033f0b24798"
server_key_version: "encryptionkey@0"
I20260924 19:17:43.128340 31036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "4b75bbfad2264396ad61062d808ad194"
format_stamp: "Formatted at 2026-09-24 19:17:43 on dist-test-slave-0lgr"
server_key: "e280a8bdf242704258d3acd1f9583d81"
server_key_iv: "c5df22e8c25933eeebd59033f0b24798"
server_key_version: "encryptionkey@0"
I20260924 19:17:43.134368 31036 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 19:17:43.138832 31051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:43.140443 31036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:17:43.140642 31036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "4b75bbfad2264396ad61062d808ad194"
format_stamp: "Formatted at 2026-09-24 19:17:43 on dist-test-slave-0lgr"
server_key: "e280a8bdf242704258d3acd1f9583d81"
server_key_iv: "c5df22e8c25933eeebd59033f0b24798"
server_key_version: "encryptionkey@0"
I20260924 19:17:43.140872 31036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:43.172107 31036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:43.177534 31036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:43.177850 31036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:43.179103 31036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:43.180723 31036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:43.180825 31036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:43.180979 31036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:43.181059 31036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:43.218719 31036 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:37037
I20260924 19:17:43.218811 31163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:37037 every 8 connection(s)
I20260924 19:17:43.220081 31036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:17:43.230324 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 31036
I20260924 19:17:43.230533 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:17:43.231094 11337 external_mini_cluster.cc:1428] Setting key c8aa8297d8685a6872f986fbd37217ab
Sep 24 19:17:43 dist-test-slave-0lgr krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277463, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.67@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:17:43.243034 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.222752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:42465 (local address 127.11.18.126:41101)
0924 19:17:43.223207 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.223214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.224219 (+ 1005us) server_negotiation.cc:407] Connection header received
0924 19:17:43.226218 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.226223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.226317 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.226444 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:43.228104 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.229010 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.230884 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.231503 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.231879 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.235947 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:43.235953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:43.235956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:43.235996 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:43.239351 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.240308 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.240312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.240315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.240473 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.241087 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.241092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.241094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.241536 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:43.241688 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.242496 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.242706 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":208,"threads_started":1}
I20260924 19:17:43.244097 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.223311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:41101 (local address 127.11.18.67:42465)
0924 19:17:43.224058 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.224100 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:43.225924 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:43.226758 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:43.226766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:43.227326 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:43.227902 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.227920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.229188 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.229192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.230724 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.230738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.231838 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.231846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.232182 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.232914 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:17:43.232939 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:43.235548 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:43.239534 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:43.239543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:43.239558 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:43.240144 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:43.240633 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:17:43.240637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:17:43.240640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:17:43.240849 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:17:43.241681 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:43.241688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:43.242069 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 19:17:43.242325 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 19:17:43.242758 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":289,"threads_started":1}
I20260924 19:17:43.246057 31164 heartbeater.cc:347] Connected to a master server at 127.11.18.126:41101
I20260924 19:17:43.246389 31164 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:43.247366 31164 heartbeater.cc:511] Master 127.11.18.126:41101 requested a full tablet report, sending...
I20260924 19:17:43.249065 30707 ts_manager.cc:194] Registered new tserver with Master: 4b75bbfad2264396ad61062d808ad194 (127.11.18.67:37037)
I20260924 19:17:43.250106 30707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.18.67@KRBTEST.COM'} at 127.11.18.67:42465
I20260924 19:17:43.260599 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:17:43.281203 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.266096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.11.18.126:41101)
0924 19:17:43.266340 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.266347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.266980 (+ 633us) server_negotiation.cc:407] Connection header received
0924 19:17:43.267141 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.267146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.267223 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.267329 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:43.269032 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.269929 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.271488 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.272031 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.272147 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.274931 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:43.274938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:43.274941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:43.274995 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:43.277884 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.278821 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.278826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.278828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.279152 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.279619 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.279627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.279632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.280044 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:43.280233 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.280712 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.280960 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 19:17:43.286393 30707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:17:43.293994 30707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:17:43.334961 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.309331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.11.18.67:37037)
0924 19:17:43.320781 (+ 11450us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.320794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.323724 (+ 2930us) server_negotiation.cc:407] Connection header received
0924 19:17:43.323885 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.323896 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.324179 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.324422 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:43.325127 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.328699 (+ 3572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.331363 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.333066 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.333212 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.333543 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.334422 (+ 879us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.334606 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11247,"thread_start_us":11024,"threads_started":1}
I20260924 19:17:43.334969 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.309098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:37037 (local address 127.0.0.1:39106)
0924 19:17:43.323344 (+ 14246us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.323373 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:43.323568 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:43.324436 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:43.324444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:43.324483 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:43.324901 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.324917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.328871 (+ 3954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.328877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.331192 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.331204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.333728 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.333736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.334026 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.334099 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 19:17:43.334236 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 19:17:43.334465 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14105,"mutex_wait_us":128,"spinlock_wait_cycles":19584,"thread_start_us":13947,"threads_started":1}
I20260924 19:17:43.334970 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.309233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.11.18.66:34001)
0924 19:17:43.324227 (+ 14994us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.324238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.324268 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:17:43.324373 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.324385 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.324712 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.324953 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:43.326466 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.328539 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.331604 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.333206 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.333373 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.333561 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.334431 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.334612 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14806,"thread_start_us":14648,"threads_started":1}
I20260924 19:17:43.335968 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.309067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:34001 (local address 127.0.0.1:53468)
0924 19:17:43.323282 (+ 14215us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.323334 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:43.323568 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:43.325304 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:43.325312 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:43.325383 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:43.325996 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.326125 (+ 129us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.328698 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.328704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.331420 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.331435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.333750 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.333757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.334196 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.334239 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:17:43.334303 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:17:43.335538 (+ 1235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13897,"mutex_wait_us":1344,"spinlock_wait_cycles":3712,"thread_start_us":13715,"threads_started":1}
I20260924 19:17:43.336524 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.324947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.0.0.1:33198)
0924 19:17:43.328615 (+ 3668us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.328640 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:43.328808 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:43.329617 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:43.329622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:43.329652 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:43.330143 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.330157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.331961 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.331968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.333508 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.333522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.335741 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.335748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.335953 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.335989 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:17:43.336082 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 19:17:43.336208 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3516,"thread_start_us":3356,"threads_started":1}
I20260924 19:17:43.337972 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.324194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.11.18.65:39351)
0924 19:17:43.328286 (+ 4092us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.328295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.328909 (+ 614us) server_negotiation.cc:407] Connection header received
0924 19:17:43.329026 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.329035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.329298 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.329606 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:43.330348 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.331825 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.333687 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.335188 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.335326 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.335576 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.337488 (+ 1912us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.337680 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3909,"thread_start_us":3737,"threads_started":1}
I20260924 19:17:43.339962 30965 tablet_service.cc:1467] Processing CreateTablet for tablet 6570b99e62ab4ec8be87cc06a066b36a (DEFAULT_TABLE table=test-table [id=62eab629ef284dc5b554674caaa16fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:43.339964 31099 tablet_service.cc:1467] Processing CreateTablet for tablet 6570b99e62ab4ec8be87cc06a066b36a (DEFAULT_TABLE table=test-table [id=62eab629ef284dc5b554674caaa16fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:43.341550 30965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6570b99e62ab4ec8be87cc06a066b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:43.341301 30831 tablet_service.cc:1467] Processing CreateTablet for tablet 6570b99e62ab4ec8be87cc06a066b36a (DEFAULT_TABLE table=test-table [id=62eab629ef284dc5b554674caaa16fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:43.341950 31099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6570b99e62ab4ec8be87cc06a066b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:43.342607 30831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6570b99e62ab4ec8be87cc06a066b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:43.355242 31183 tablet_bootstrap.cc:492] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: Bootstrap starting.
I20260924 19:17:43.358439 31185 tablet_bootstrap.cc:492] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Bootstrap starting.
I20260924 19:17:43.359071 31184 tablet_bootstrap.cc:492] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Bootstrap starting.
I20260924 19:17:43.359757 31183 tablet_bootstrap.cc:654] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:43.361255 31183 log.cc:824] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:43.362411 31185 tablet_bootstrap.cc:654] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:43.363209 31184 tablet_bootstrap.cc:654] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:43.363646 31185 log.cc:824] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:43.364904 31184 log.cc:824] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:43.368599 31185 tablet_bootstrap.cc:492] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: No bootstrap required, opened a new log
I20260924 19:17:43.368654 31184 tablet_bootstrap.cc:492] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: No bootstrap required, opened a new log
I20260924 19:17:43.368654 31183 tablet_bootstrap.cc:492] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: No bootstrap required, opened a new log
I20260924 19:17:43.368906 31185 ts_tablet_manager.cc:1397] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 19:17:43.368935 31184 ts_tablet_manager.cc:1397] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 19:17:43.368935 31183 ts_tablet_manager.cc:1397] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260924 19:17:43.377434 31185 raft_consensus.cc:348] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.377883 31185 raft_consensus.cc:374] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:43.378000 31185 raft_consensus.cc:729] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b75bbfad2264396ad61062d808ad194, State: Initialized, Role: FOLLOWER
I20260924 19:17:43.378788 31185 consensus_queue.cc:260] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.379995 31164 heartbeater.cc:503] Master 127.11.18.126:41101 was elected leader, sending a full tablet report...
I20260924 19:17:43.380743 31185 ts_tablet_manager.cc:1428] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 19:17:43.381410 31183 raft_consensus.cc:348] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.381400 31184 raft_consensus.cc:348] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.381896 31184 raft_consensus.cc:374] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:43.382009 31184 raft_consensus.cc:729] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6d10ba02a54913801f0f00409376cc, State: Initialized, Role: FOLLOWER
I20260924 19:17:43.382130 31183 raft_consensus.cc:374] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:43.382242 31183 raft_consensus.cc:729] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3bf529199f4420b2ea5e41353dd42c, State: Initialized, Role: FOLLOWER
I20260924 19:17:43.382750 31184 consensus_queue.cc:260] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.382866 31183 consensus_queue.cc:260] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.384003 31030 heartbeater.cc:503] Master 127.11.18.126:41101 was elected leader, sending a full tablet report...
I20260924 19:17:43.385123 31184 ts_tablet_manager.cc:1428] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260924 19:17:43.387822 31183 ts_tablet_manager.cc:1428] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: Time spent starting tablet: real 0.019s user 0.012s sys 0.008s
W20260924 19:17:43.401516 31031 tablet.cc:2369] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:17:43.471936 31165 tablet.cc:2369] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:43.477193 31190 raft_consensus.cc:482] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:43.477633 31190 raft_consensus.cc:504] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.479590 31190 leader_election.cc:288] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b75bbfad2264396ad61062d808ad194 (127.11.18.67:37037), bc3bf529199f4420b2ea5e41353dd42c (127.11.18.65:39351)
I20260924 19:17:43.487654 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.480118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:60385 (local address 127.11.18.67:37037)
0924 19:17:43.480355 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.480359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.481355 (+ 996us) server_negotiation.cc:407] Connection header received
0924 19:17:43.481696 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.481702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.481787 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.481898 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:43.482761 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.483505 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.485806 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.486601 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.486787 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.486887 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.487251 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.487382 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:17:43.487708 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.480331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:50293 (local address 127.11.18.65:39351)
0924 19:17:43.480534 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.480539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.481696 (+ 1157us) server_negotiation.cc:407] Connection header received
0924 19:17:43.481837 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.481843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.481927 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.482023 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:43.482745 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.483612 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.485774 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.486694 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.486877 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.486955 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.487286 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.487423 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:17:43.487831 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.480700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.11.18.66:50293)
0924 19:17:43.481611 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.481626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:43.481733 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:43.482057 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:43.482064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:43.482094 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:43.482544 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.482552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.483801 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.483809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.485595 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.485610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.486937 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.486946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.487100 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.487123 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 19:17:43.487236 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:17:43.487365 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":26,"thread_start_us":541,"threads_started":1}
I20260924 19:17:43.487891 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.480142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:37037 (local address 127.11.18.66:60385)
0924 19:17:43.481204 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.481221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:43.481464 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:43.481965 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:43.481969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:43.482030 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:43.482517 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.482537 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.483655 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.483664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.485611 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:43.485626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.486862 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.486874 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:43.487074 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.487104 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:17:43.487198 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 19:17:43.487377 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":79,"thread_start_us":674,"threads_started":1}
I20260924 19:17:43.489153 31119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6570b99e62ab4ec8be87cc06a066b36a" candidate_uuid: "2c6d10ba02a54913801f0f00409376cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b75bbfad2264396ad61062d808ad194" is_pre_election: true
I20260924 19:17:43.489151 30851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6570b99e62ab4ec8be87cc06a066b36a" candidate_uuid: "2c6d10ba02a54913801f0f00409376cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3bf529199f4420b2ea5e41353dd42c" is_pre_election: true
I20260924 19:17:43.489800 31119 raft_consensus.cc:2454] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6d10ba02a54913801f0f00409376cc in term 0.
I20260924 19:17:43.489814 30851 raft_consensus.cc:2454] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6d10ba02a54913801f0f00409376cc in term 0.
I20260924 19:17:43.490545 30919 leader_election.cc:302] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6d10ba02a54913801f0f00409376cc, 4b75bbfad2264396ad61062d808ad194; no voters:
I20260924 19:17:43.490932 31190 raft_consensus.cc:2781] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:17:43.491151 31190 raft_consensus.cc:482] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:17:43.491274 31190 raft_consensus.cc:3037] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:43.494088 31190 raft_consensus.cc:504] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.494731 31190 leader_election.cc:288] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [CANDIDATE]: Term 1 election: Requested vote from peers 4b75bbfad2264396ad61062d808ad194 (127.11.18.67:37037), bc3bf529199f4420b2ea5e41353dd42c (127.11.18.65:39351)
I20260924 19:17:43.495415 31119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6570b99e62ab4ec8be87cc06a066b36a" candidate_uuid: "2c6d10ba02a54913801f0f00409376cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b75bbfad2264396ad61062d808ad194"
I20260924 19:17:43.495406 30851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6570b99e62ab4ec8be87cc06a066b36a" candidate_uuid: "2c6d10ba02a54913801f0f00409376cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3bf529199f4420b2ea5e41353dd42c"
I20260924 19:17:43.495667 30851 raft_consensus.cc:3037] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:43.495667 31119 raft_consensus.cc:3037] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:43.499079 30851 raft_consensus.cc:2454] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6d10ba02a54913801f0f00409376cc in term 1.
I20260924 19:17:43.499078 31119 raft_consensus.cc:2454] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6d10ba02a54913801f0f00409376cc in term 1.
I20260924 19:17:43.499590 30919 leader_election.cc:302] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6d10ba02a54913801f0f00409376cc, 4b75bbfad2264396ad61062d808ad194; no voters:
I20260924 19:17:43.499926 31190 raft_consensus.cc:2781] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:43.500428 31190 raft_consensus.cc:686] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 1 LEADER]: Becoming Leader. State: Replica: 2c6d10ba02a54913801f0f00409376cc, State: Running, Role: LEADER
I20260924 19:17:43.501021 31190 consensus_queue.cc:237] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } }
I20260924 19:17:43.505942 30707 catalog_manager.cc:5495] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c6d10ba02a54913801f0f00409376cc (127.11.18.66). New cstate: current_term: 1 leader_uuid: "2c6d10ba02a54913801f0f00409376cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6d10ba02a54913801f0f00409376cc" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 34001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 } health_report { overall_health: UNKNOWN } } }
I20260924 19:17:43.562486 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.556436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.11.18.66:34001)
0924 19:17:43.556683 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.556689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.556791 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:17:43.557152 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.557156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.557247 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.557548 (+ 301us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:17:43.557915 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.558631 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.560021 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.560507 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.560647 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.561169 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:17:43.561764 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:17:43.561924 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.562035 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.562121 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:17:43.570883 31119 raft_consensus.cc:1261] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 1 FOLLOWER]: Refusing update from remote peer 2c6d10ba02a54913801f0f00409376cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:43.570880 30851 raft_consensus.cc:1261] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 1 FOLLOWER]: Refusing update from remote peer 2c6d10ba02a54913801f0f00409376cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:43.572242 31190 consensus_queue.cc:1035] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc3bf529199f4420b2ea5e41353dd42c" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:43.572613 31194 consensus_queue.cc:1035] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b75bbfad2264396ad61062d808ad194" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:17:43.574504 30897 tablet.cc:2369] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:43.603410 30706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:43.603768 30706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:43.607900 30706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f97368e726b4fb6a68ae628beae2220: Sending DeleteTablet for 3 replicas of tablet 6570b99e62ab4ec8be87cc06a066b36a
I20260924 19:17:43.609264 31099 tablet_service.cc:1514] Processing DeleteTablet for tablet 6570b99e62ab4ec8be87cc06a066b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:43 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:39106
I20260924 19:17:43.609397 30965 tablet_service.cc:1514] Processing DeleteTablet for tablet 6570b99e62ab4ec8be87cc06a066b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:43 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:53468
I20260924 19:17:43.610529 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 30768
I20260924 19:17:43.610544 30831 tablet_service.cc:1514] Processing DeleteTablet for tablet 6570b99e62ab4ec8be87cc06a066b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:43 UTC) from {username='kudu', principal='kudu/127.11.18.126@KRBTEST.COM'} at 127.0.0.1:33198
I20260924 19:17:43.636987 31209 tablet_replica.cc:330] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: stopping tablet replica
I20260924 19:17:43.637790 31207 tablet_replica.cc:330] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: stopping tablet replica
Sep 24 19:17:43 dist-test-slave-0lgr krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.65@KRBTEST.COM
I20260924 19:17:43.638388 31207 raft_consensus.cc:2229] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:43.638998 31209 raft_consensus.cc:2229] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:43.639602 31209 raft_consensus.cc:2258] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:43.639669 31208 tablet_replica.cc:330] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: stopping tablet replica
I20260924 19:17:43.640177 31207 raft_consensus.cc:2258] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:43.640200 31208 raft_consensus.cc:2229] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:17:43.640987 31208 raft_consensus.cc:2258] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:43.643568 31207 ts_tablet_manager.cc:1905] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:43.645114 31208 ts_tablet_manager.cc:1905] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:43.645830 31209 ts_tablet_manager.cc:1905] T 6570b99e62ab4ec8be87cc06a066b36a P bc3bf529199f4420b2ea5e41353dd42c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:43.649109 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.614301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.11.18.65:39351)
0924 19:17:43.614543 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.614562 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.623139 (+ 8577us) server_negotiation.cc:407] Connection header received
0924 19:17:43.623760 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.623769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.623898 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.623992 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:43.626800 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.628392 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.631055 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.632008 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.632136 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.639868 (+ 7732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:43.639887 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:43.639906 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:43.639960 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:43.644829 (+ 4869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.645957 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.645967 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.645971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.646159 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.646748 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.646754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.646757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.647292 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:43.647652 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.648355 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.648655 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:17:43.650731 30891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:43.662706 31207 ts_tablet_manager.cc:1918] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:43.663136 31208 ts_tablet_manager.cc:1918] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:43.663565 31208 log.cc:1196] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/6570b99e62ab4ec8be87cc06a066b36a
I20260924 19:17:43.664569 31208 ts_tablet_manager.cc:1939] T 6570b99e62ab4ec8be87cc06a066b36a P 2c6d10ba02a54913801f0f00409376cc: Deleting consensus metadata
I20260924 19:17:43.667678 30693 catalog_manager.cc:4841] TS 2c6d10ba02a54913801f0f00409376cc (127.11.18.66:34001): tablet 6570b99e62ab4ec8be87cc06a066b36a (table test-table [id=62eab629ef284dc5b554674caaa16fd6]) successfully deleted
I20260924 19:17:43.668556 31207 log.cc:1196] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/6570b99e62ab4ec8be87cc06a066b36a
I20260924 19:17:43.670291 31207 ts_tablet_manager.cc:1939] T 6570b99e62ab4ec8be87cc06a066b36a P 4b75bbfad2264396ad61062d808ad194: Deleting consensus metadata
I20260924 19:17:43.673159 30692 catalog_manager.cc:4841] TS 4b75bbfad2264396ad61062d808ad194 (127.11.18.67:37037): tablet 6570b99e62ab4ec8be87cc06a066b36a (table test-table [id=62eab629ef284dc5b554674caaa16fd6]) successfully deleted
I20260924 19:17:43.768872 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 30768
I20260924 19:17:43.787578 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 30902
W20260924 19:17:43.787606 30693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.18.65:39351 (error 108)
I20260924 19:17:43.790906 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.790318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.0.0.1:33218)
0924 19:17:43.790566 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.790705 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:17:43.791230 30693 catalog_manager.cc:4584] TS bc3bf529199f4420b2ea5e41353dd42c (127.11.18.65:39351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6570b99e62ab4ec8be87cc06a066b36a: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39351: connect: Connection refused (error 111)
Sep 24 19:17:43 dist-test-slave-0lgr krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.66@KRBTEST.COM
I20260924 19:17:43.806723 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.788665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.11.18.66:34001)
0924 19:17:43.788865 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.788871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.788893 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:17:43.788963 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.788969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.789079 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.789209 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:43.790902 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.791913 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.793217 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.793725 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.794051 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.798748 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:43.798771 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:43.798793 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:43.798846 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:43.803130 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.804076 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.804091 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.804095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.804261 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.804707 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.804712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.804715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.805234 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:43.805704 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.806051 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.806309 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:17:43.807578 31025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:43.845192 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.844606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.0.0.1:33220)
0924 19:17:43.844896 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.845016 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 19:17:43.846392 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.846024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.0.0.1:33228)
0924 19:17:43.846181 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.846259 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:17:43.907291 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.906713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.0.0.1:33238)
0924 19:17:43.907003 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.907106 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 19:17:43.908305 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.907967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.0.0.1:33246)
0924 19:17:43.908116 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.908191 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 19:17:43.930159 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 30902
I20260924 19:17:43.950155 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31036
Sep 24 19:17:43 dist-test-slave-0lgr krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.67@KRBTEST.COM
I20260924 19:17:43.970763 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.950927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.11.18.67:37037)
0924 19:17:43.951115 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:43.951119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:43.951138 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:17:43.951284 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:43.951288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:43.951353 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:43.951440 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:43.952640 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.953470 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.954855 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:43.955238 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:43.955328 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:43.964832 (+ 9504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:43.964849 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:43.964865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:43.964909 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:43.967800 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.968515 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.968522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.968525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.968674 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:43.969085 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:43.969088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:43.969090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:43.969538 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:43.969759 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:43.970234 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 19:17:43.970419 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:17:43.971467 31159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:43.989748 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.989103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.0.0.1:33250)
0924 19:17:43.989445 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.989544 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 19:17:43.991014 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:43.990542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:39351 (local address 127.0.0.1:33262)
0924 19:17:43.990782 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:43.990881 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.65:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 19:17:44.065794 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 31036
I20260924 19:17:44.083281 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 30675
I20260924 19:17:44.084543 30737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:44.200536 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 30675
2026-09-24T19:17:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4721 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T19:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:44Z Disabled control of system clock
I20260924 19:17:44.283854 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:44929
--webserver_interface=127.11.18.126
--webserver_port=0
--builtin_ntp_servers=127.11.18.84:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:44929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:17:44.630366 31219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:44.630748 31219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:44.630904 31219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:44.638566 31219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:44.638732 31219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:44.638819 31219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:44.638926 31219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:44.639010 31219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:44.648708 31219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:44929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:44.650132 31219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:44.652034 31219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:44.664913 31224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:44.664915 31225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:44.665001 31227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:44.666396 31219 server_base.cc:1034] running on GCE node
I20260924 19:17:44.667455 31219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:44.669034 31219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:44.670326 31219 hybrid_clock.cc:648] HybridClock initialized: now 1790277464670266 us; error 40 us; skew 500 ppm
I20260924 19:17:44.670769 31219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:44.673547 31219 webserver.cc:460] Webserver started at http://127.11.18.126:33487/ using document root <none> and password file <none>
I20260924 19:17:44.674393 31219 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:44.674522 31219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:44.674867 31219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:44.678197 31219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "8ceacbf998784f1ca7ce2fb117615dd0"
format_stamp: "Formatted at 2026-09-24 19:17:44 on dist-test-slave-0lgr"
server_key: "4d36a964c356c1e97a622efaa44cd0a5"
server_key_iv: "c70d5097cb5256abe6d42a826fffa534"
server_key_version: "encryptionkey@0"
I20260924 19:17:44.679081 31219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "8ceacbf998784f1ca7ce2fb117615dd0"
format_stamp: "Formatted at 2026-09-24 19:17:44 on dist-test-slave-0lgr"
server_key: "4d36a964c356c1e97a622efaa44cd0a5"
server_key_iv: "c70d5097cb5256abe6d42a826fffa534"
server_key_version: "encryptionkey@0"
I20260924 19:17:44.684790 31219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 19:17:44.688712 31233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:44.690404 31219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:44.690623 31219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "8ceacbf998784f1ca7ce2fb117615dd0"
format_stamp: "Formatted at 2026-09-24 19:17:44 on dist-test-slave-0lgr"
server_key: "4d36a964c356c1e97a622efaa44cd0a5"
server_key_iv: "c70d5097cb5256abe6d42a826fffa534"
server_key_version: "encryptionkey@0"
I20260924 19:17:44.690857 31219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:44.723994 31219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:44.724922 31219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:44.725235 31219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:44.745770 31219 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:44929
I20260924 19:17:44.745796 31284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:44929 every 8 connection(s)
I20260924 19:17:44.747231 31219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:44.751005 31285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:44.751046 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 31219
I20260924 19:17:44.751296 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:44.751776 11337 external_mini_cluster.cc:1428] Setting key 671c834ee97cebc3504804d08e66fa8f
I20260924 19:17:44.765003 31285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0: Bootstrap starting.
I20260924 19:17:44.767288 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:44.753984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.11.18.126:44929)
0924 19:17:44.755133 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:44.755144 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:44.755186 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:17:44.757285 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:44.757309 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:44.757926 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:44.758293 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:44.759090 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:44.760406 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:44.762490 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:44.763260 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:44.763401 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:44.764707 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:44.764721 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:44.764736 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:44.765438 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:44.765531 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:44.765557 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:44.765666 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:44.765906 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:17:44.766331 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":450,"threads_started":1}
I20260924 19:17:44.768539 31285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:44.769656 31285 log.cc:824] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:44.772584 31285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0: No bootstrap required, opened a new log
I20260924 19:17:44.781517 31285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceacbf998784f1ca7ce2fb117615dd0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44929 } }
I20260924 19:17:44.781937 31285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:44.782017 31285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ceacbf998784f1ca7ce2fb117615dd0, State: Initialized, Role: FOLLOWER
I20260924 19:17:44.782648 31285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceacbf998784f1ca7ce2fb117615dd0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44929 } }
I20260924 19:17:44.782924 31285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:44.783073 31285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:44.783303 31285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:44.786057 31285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceacbf998784f1ca7ce2fb117615dd0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44929 } }
I20260924 19:17:44.786662 31285 leader_election.cc:302] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ceacbf998784f1ca7ce2fb117615dd0; no voters:
I20260924 19:17:44.787367 31285 leader_election.cc:288] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:44.787477 31290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:44.788270 31290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [term 1 LEADER]: Becoming Leader. State: Replica: 8ceacbf998784f1ca7ce2fb117615dd0, State: Running, Role: LEADER
I20260924 19:17:44.788851 31290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceacbf998784f1ca7ce2fb117615dd0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44929 } }
I20260924 19:17:44.789765 31285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:44.791885 31292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ceacbf998784f1ca7ce2fb117615dd0. Latest consensus state: current_term: 1 leader_uuid: "8ceacbf998784f1ca7ce2fb117615dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceacbf998784f1ca7ce2fb117615dd0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44929 } } }
I20260924 19:17:44.791848 31291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ceacbf998784f1ca7ce2fb117615dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceacbf998784f1ca7ce2fb117615dd0" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 44929 } } }
I20260924 19:17:44.792321 31292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:44.792318 31291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:44.795506 31296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:44.801663 31296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:44.811319 31296 catalog_manager.cc:1347] Generated new cluster ID: e0bb9b5e28034cf49946bab6e12ce974
I20260924 19:17:44.811460 31296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:44.836217 31296 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:44.837797 31296 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:44.861783 31296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0: Generated new TSK 0
I20260924 19:17:44.862690 31296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:44.880486 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.65:0
--local_ip_for_outbound_sockets=127.11.18.65
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44929
--builtin_ntp_servers=127.11.18.84:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:17:45.221230 31309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:45.221622 31309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:45.221714 31309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:45.229151 31309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:45.229254 31309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:45.229424 31309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.65
I20260924 19:17:45.238813 31309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.18.65
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.65
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:45.240054 31309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:45.241714 31309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:45.255704 31314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:45.255600 31315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:45.255777 31317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:45.256611 31309 server_base.cc:1034] running on GCE node
I20260924 19:17:45.257257 31309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:45.258447 31309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:45.259670 31309 hybrid_clock.cc:648] HybridClock initialized: now 1790277465259599 us; error 60 us; skew 500 ppm
I20260924 19:17:45.260043 31309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:45.262516 31309 webserver.cc:460] Webserver started at http://127.11.18.65:38223/ using document root <none> and password file <none>
I20260924 19:17:45.263343 31309 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:45.263481 31309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:45.263818 31309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:45.266903 31309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/data/instance:
uuid: "472fde66e6c44ce2861c402ae7230b87"
format_stamp: "Formatted at 2026-09-24 19:17:45 on dist-test-slave-0lgr"
server_key: "67bb185ae739651882d37d3257188542"
server_key_iv: "f6a235714c48ae832c1051fd7255041a"
server_key_version: "encryptionkey@0"
I20260924 19:17:45.267805 31309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance:
uuid: "472fde66e6c44ce2861c402ae7230b87"
format_stamp: "Formatted at 2026-09-24 19:17:45 on dist-test-slave-0lgr"
server_key: "67bb185ae739651882d37d3257188542"
server_key_iv: "f6a235714c48ae832c1051fd7255041a"
server_key_version: "encryptionkey@0"
I20260924 19:17:45.273432 31309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:17:45.277087 31323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:45.278700 31309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:45.278914 31309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/wal
uuid: "472fde66e6c44ce2861c402ae7230b87"
format_stamp: "Formatted at 2026-09-24 19:17:45 on dist-test-slave-0lgr"
server_key: "67bb185ae739651882d37d3257188542"
server_key_iv: "f6a235714c48ae832c1051fd7255041a"
server_key_version: "encryptionkey@0"
I20260924 19:17:45.279161 31309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:45.290472 31309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:45.291363 31309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:45.291682 31309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:45.292711 31309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:45.294445 31309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:45.294570 31309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:45.294703 31309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:45.294811 31309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:45.331476 31309 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.65:46171
I20260924 19:17:45.331535 31435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.65:46171 every 8 connection(s)
I20260924 19:17:45.332911 31309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/data/info.pb
I20260924 19:17:45.336320 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 31309
I20260924 19:17:45.336519 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/wal/instance
I20260924 19:17:45.337024 11337 external_mini_cluster.cc:1428] Setting key 4d913270cd134f32a8f957187d32af68
I20260924 19:17:45.339884 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.66:0
--local_ip_for_outbound_sockets=127.11.18.66
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44929
--builtin_ntp_servers=127.11.18.84:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:17:45.347795 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:45.335863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:50601 (local address 127.11.18.126:44929)
0924 19:17:45.336500 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:45.336508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:45.337892 (+ 1384us) server_negotiation.cc:407] Connection header received
0924 19:17:45.339740 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:45.339746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:45.339829 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:45.339958 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:45.341169 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:45.342426 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:45.343974 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:45.344712 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:45.344852 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:45.346713 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:45.346722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:45.346729 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:45.346757 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:45.346813 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:45.346832 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:45.346939 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:45.347199 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 19:17:45.347324 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":290,"threads_started":1}
I20260924 19:17:45.348869 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:45.336578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:44929 (local address 127.11.18.65:50601)
0924 19:17:45.337720 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:45.337771 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:45.339471 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:45.340287 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:45.340295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:45.340393 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:45.340978 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:45.340995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:45.342573 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:45.342577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:45.343825 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:45.343837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:45.344978 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:45.344984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:45.345351 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:45.346299 (+ 948us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:45.346305 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:45.346362 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:45.346475 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:45.346946 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:45.346959 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:45.346970 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:17:45.347160 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 19:17:45.347610 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":360,"threads_started":1}
I20260924 19:17:45.350654 31436 heartbeater.cc:347] Connected to a master server at 127.11.18.126:44929
I20260924 19:17:45.351025 31436 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:45.351987 31436 heartbeater.cc:511] Master 127.11.18.126:44929 requested a full tablet report, sending...
I20260924 19:17:45.354753 31250 ts_manager.cc:194] Registered new tserver with Master: 472fde66e6c44ce2861c402ae7230b87 (127.11.18.65:46171)
I20260924 19:17:45.356389 31250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.65:50601
W20260924 19:17:45.691820 31440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:45.692159 31440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:45.692310 31440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:45.699982 31440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:45.700120 31440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:45.700323 31440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.66
I20260924 19:17:45.710048 31440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.18.66
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.66
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:45.711376 31440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:45.713151 31440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:45.727092 31446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:45.727330 31445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:45.727373 31448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:45.728880 31440 server_base.cc:1034] running on GCE node
I20260924 19:17:45.729750 31440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:45.731007 31440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:45.732290 31440 hybrid_clock.cc:648] HybridClock initialized: now 1790277465732229 us; error 54 us; skew 500 ppm
I20260924 19:17:45.732682 31440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:45.735316 31440 webserver.cc:460] Webserver started at http://127.11.18.66:43243/ using document root <none> and password file <none>
I20260924 19:17:45.736155 31440 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:45.736307 31440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:45.736650 31440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:45.740087 31440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/data/instance:
uuid: "2c2bba213d2f47d0aefd4aebff698c6b"
format_stamp: "Formatted at 2026-09-24 19:17:45 on dist-test-slave-0lgr"
server_key: "0de685a44f62251ffe0f65f4bbe3a596"
server_key_iv: "3c58fed8e7342e3f76a40625a6b55631"
server_key_version: "encryptionkey@0"
I20260924 19:17:45.740975 31440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance:
uuid: "2c2bba213d2f47d0aefd4aebff698c6b"
format_stamp: "Formatted at 2026-09-24 19:17:45 on dist-test-slave-0lgr"
server_key: "0de685a44f62251ffe0f65f4bbe3a596"
server_key_iv: "3c58fed8e7342e3f76a40625a6b55631"
server_key_version: "encryptionkey@0"
I20260924 19:17:45.746783 31440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:17:45.750835 31454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:45.752344 31440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260924 19:17:45.752543 31440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/wal
uuid: "2c2bba213d2f47d0aefd4aebff698c6b"
format_stamp: "Formatted at 2026-09-24 19:17:45 on dist-test-slave-0lgr"
server_key: "0de685a44f62251ffe0f65f4bbe3a596"
server_key_iv: "3c58fed8e7342e3f76a40625a6b55631"
server_key_version: "encryptionkey@0"
I20260924 19:17:45.752784 31440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:45.769240 31440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:45.770323 31440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:45.770665 31440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:45.771777 31440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:45.773484 31440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:45.773612 31440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:45.773772 31440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:45.773880 31440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:45.811842 31440 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.66:36791
I20260924 19:17:45.811939 31566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.66:36791 every 8 connection(s)
I20260924 19:17:45.813375 31440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/data/info.pb
I20260924 19:17:45.820304 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 31440
I20260924 19:17:45.820537 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/wal/instance
I20260924 19:17:45.821018 11337 external_mini_cluster.cc:1428] Setting key 27ccaf8e65480f35d4254fde91c98fbc
I20260924 19:17:45.823895 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.18.67:0
--local_ip_for_outbound_sockets=127.11.18.67
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44929
--builtin_ntp_servers=127.11.18.84:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:17:45.827732 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:45.816286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.66:50281 (local address 127.11.18.126:44929)
0924 19:17:45.816499 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:45.816505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:45.818137 (+ 1632us) server_negotiation.cc:407] Connection header received
0924 19:17:45.820076 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:45.820085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:45.820183 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:45.820310 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:45.821779 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:45.822603 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:45.824494 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:45.824948 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:45.825099 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:45.826742 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:45.826748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:45.826752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:45.826779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:45.826814 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:45.826828 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:45.826924 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:45.827254 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 19:17:45.827382 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:17:45.828903 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:45.816813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:44929 (local address 127.11.18.66:50281)
0924 19:17:45.817971 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:45.818018 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:45.819752 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:45.820632 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:45.820647 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:45.820756 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:45.821580 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:45.821600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:45.822777 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:45.822784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:45.824325 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:45.824341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:45.825207 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:45.825216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:45.825550 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:45.826326 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:45.826329 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:45.826384 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:45.826487 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:45.827011 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:45.827021 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:45.827028 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:17:45.827225 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 19:17:45.827669 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":320,"threads_started":1}
I20260924 19:17:45.830845 31567 heartbeater.cc:347] Connected to a master server at 127.11.18.126:44929
I20260924 19:17:45.831274 31567 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:45.832194 31567 heartbeater.cc:511] Master 127.11.18.126:44929 requested a full tablet report, sending...
I20260924 19:17:45.833982 31250 ts_manager.cc:194] Registered new tserver with Master: 2c2bba213d2f47d0aefd4aebff698c6b (127.11.18.66:36791)
I20260924 19:17:45.835237 31250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.66:50281
W20260924 19:17:46.171168 31570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:46.171468 31570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:46.171555 31570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:46.179044 31570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:46.179162 31570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:46.179272 31570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.18.67
I20260924 19:17:46.188661 31570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.18.67
--webserver_port=0
--tserver_master_addrs=127.11.18.126:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.18.67
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:46.190231 31570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:46.192157 31570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:46.207463 31576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:46.207589 31575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:46.207561 31578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:46.211210 31570 server_base.cc:1034] running on GCE node
I20260924 19:17:46.211935 31570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:46.212980 31570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:46.214226 31570 hybrid_clock.cc:648] HybridClock initialized: now 1790277466214160 us; error 61 us; skew 500 ppm
I20260924 19:17:46.214567 31570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:17:46.216970 31570 webserver.cc:460] Webserver started at http://127.11.18.67:36115/ using document root <none> and password file <none>
I20260924 19:17:46.217896 31570 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:46.218029 31570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:46.218340 31570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:46.221199 31570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/data/instance:
uuid: "ec78a7b6a25144939d761361d81c2b62"
format_stamp: "Formatted at 2026-09-24 19:17:46 on dist-test-slave-0lgr"
server_key: "03e58f7ac14a9c4125b2724c1f5513e1"
server_key_iv: "136d68bde708ef2dc992604d2176a52f"
server_key_version: "encryptionkey@0"
I20260924 19:17:46.222086 31570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance:
uuid: "ec78a7b6a25144939d761361d81c2b62"
format_stamp: "Formatted at 2026-09-24 19:17:46 on dist-test-slave-0lgr"
server_key: "03e58f7ac14a9c4125b2724c1f5513e1"
server_key_iv: "136d68bde708ef2dc992604d2176a52f"
server_key_version: "encryptionkey@0"
I20260924 19:17:46.227393 31570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 19:17:46.231228 31584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:46.232810 31570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:17:46.232957 31570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/wal
uuid: "ec78a7b6a25144939d761361d81c2b62"
format_stamp: "Formatted at 2026-09-24 19:17:46 on dist-test-slave-0lgr"
server_key: "03e58f7ac14a9c4125b2724c1f5513e1"
server_key_iv: "136d68bde708ef2dc992604d2176a52f"
server_key_version: "encryptionkey@0"
I20260924 19:17:46.233150 31570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:46.261106 31570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:46.262100 31570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:46.262384 31570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:46.263409 31570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:17:46.265051 31570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:17:46.265146 31570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:46.265226 31570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:17:46.265290 31570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:46.303120 31570 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.67:39559
I20260924 19:17:46.303241 31696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.67:39559 every 8 connection(s)
I20260924 19:17:46.304478 31570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/data/info.pb
I20260924 19:17:46.312899 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 31570
I20260924 19:17:46.313081 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/wal/instance
I20260924 19:17:46.313653 11337 external_mini_cluster.cc:1428] Setting key 29cfa550eb60b66b0f985866357f39cb
I20260924 19:17:46.318907 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.307308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:42461 (local address 127.11.18.126:44929)
0924 19:17:46.307545 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.307552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.309010 (+ 1458us) server_negotiation.cc:407] Connection header received
0924 19:17:46.310879 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.310886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.310971 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.311072 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:46.312215 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.313473 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.315728 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.316343 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.316490 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.318074 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:46.318079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:46.318082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:46.318100 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:46.318132 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:46.318144 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:46.318218 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.318557 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.318642 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:17:46.319943 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.307831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:44929 (local address 127.11.18.67:42461)
0924 19:17:46.308776 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:46.308819 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:46.310596 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:46.311333 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:46.311342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:46.311437 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:17:46.312037 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.312054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.313664 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.313669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.315564 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.315577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.316593 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.316599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.316940 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.317714 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:17:46.317717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:17:46.317761 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:17:46.317859 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:17:46.318240 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:17:46.318249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:17:46.318255 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:17:46.318458 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 19:17:46.318773 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":289,"threads_started":1}
I20260924 19:17:46.321570 31697 heartbeater.cc:347] Connected to a master server at 127.11.18.126:44929
I20260924 19:17:46.321900 31697 heartbeater.cc:464] Registering TS with master...
I20260924 19:17:46.322674 31697 heartbeater.cc:511] Master 127.11.18.126:44929 requested a full tablet report, sending...
I20260924 19:17:46.324337 31250 ts_manager.cc:194] Registered new tserver with Master: ec78a7b6a25144939d761361d81c2b62 (127.11.18.67:39559)
I20260924 19:17:46.325243 31250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.18.67:42461
I20260924 19:17:46.329860 11337 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:17:46.342031 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.334550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.11.18.126:44929)
0924 19:17:46.334747 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.334754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.335879 (+ 1125us) server_negotiation.cc:407] Connection header received
0924 19:17:46.336014 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.336020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.336124 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.336254 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:46.336780 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.337791 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.339421 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.339898 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.340004 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.341224 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:46.341230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:46.341234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:46.341254 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:46.341291 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:46.341304 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:46.341477 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.341643 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.341725 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:17:46.346639 31250 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:17:46.354001 31250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:17:46.360412 31436 heartbeater.cc:503] Master 127.11.18.126:44929 was elected leader, sending a full tablet report...
I20260924 19:17:46.377871 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.367128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.11.18.66:36791)
0924 19:17:46.367680 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.367689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.368427 (+ 738us) server_negotiation.cc:407] Connection header received
0924 19:17:46.368818 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.368826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.369091 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.369282 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:46.370223 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.371490 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.374294 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.375792 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.375920 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.376183 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.377268 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.377500 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":269,"threads_started":1}
I20260924 19:17:46.378161 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.366967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:36791 (local address 127.0.0.1:37110)
0924 19:17:46.368282 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:46.368328 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:46.368556 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:46.369596 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:46.369604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:46.369644 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:46.370040 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.370053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.371614 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.371622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.374052 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.374066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.376707 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.376718 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.376991 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.377054 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 19:17:46.377195 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 19:17:46.377507 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":121,"thread_start_us":1981,"threads_started":1}
I20260924 19:17:46.378636 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.368623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:46171 (local address 127.0.0.1:52066)
0924 19:17:46.369501 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:46.369516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:46.369605 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:46.370715 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:46.370719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:46.370744 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:46.371146 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.371158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.373189 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.373194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.375488 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.375510 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.377750 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.377758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.377999 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.378031 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:17:46.378099 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:17:46.378236 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":36,"thread_start_us":772,"threads_started":1}
I20260924 19:17:46.378700 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.367767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.11.18.67:39559)
0924 19:17:46.368587 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.368595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.369845 (+ 1250us) server_negotiation.cc:407] Connection header received
0924 19:17:46.369974 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.369980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.370196 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.370406 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:46.370976 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.372773 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.374852 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.375745 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.375897 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.376217 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.378161 (+ 1944us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.378323 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":421,"threads_started":1}
I20260924 19:17:46.378326 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.367089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:39559 (local address 127.0.0.1:48184)
0924 19:17:46.369233 (+ 2144us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:46.369255 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:46.369427 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:46.370402 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:46.370406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:46.370431 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:46.370778 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.370796 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.372943 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.372948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.374655 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.374667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.376597 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.376604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.376880 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.377054 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 19:17:46.377195 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 19:17:46.377507 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1996,"mutex_wait_us":149,"thread_start_us":697,"threads_started":1}
I20260924 19:17:46.379356 31714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.369193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.11.18.65:46171)
0924 19:17:46.370111 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.370119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.370143 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:17:46.370233 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.370241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.370484 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.371349 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:46.371597 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.373046 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.376204 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.377527 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.378428 (+ 901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.378722 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.378797 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.379090 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":599,"threads_started":1}
I20260924 19:17:46.382824 31502 tablet_service.cc:1467] Processing CreateTablet for tablet f7e8b98651f64a68a7f6858fdf4c6d21 (DEFAULT_TABLE table=test-table [id=1e6b335191614538bdaa8e7004a46158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:46.383940 31371 tablet_service.cc:1467] Processing CreateTablet for tablet f7e8b98651f64a68a7f6858fdf4c6d21 (DEFAULT_TABLE table=test-table [id=1e6b335191614538bdaa8e7004a46158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:46.384588 31502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e8b98651f64a68a7f6858fdf4c6d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:46.384160 31632 tablet_service.cc:1467] Processing CreateTablet for tablet f7e8b98651f64a68a7f6858fdf4c6d21 (DEFAULT_TABLE table=test-table [id=1e6b335191614538bdaa8e7004a46158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:17:46.385457 31371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e8b98651f64a68a7f6858fdf4c6d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:46.385666 31632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e8b98651f64a68a7f6858fdf4c6d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:46.397933 31715 tablet_bootstrap.cc:492] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Bootstrap starting.
I20260924 19:17:46.400292 31716 tablet_bootstrap.cc:492] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Bootstrap starting.
I20260924 19:17:46.402412 31717 tablet_bootstrap.cc:492] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Bootstrap starting.
I20260924 19:17:46.403199 31715 tablet_bootstrap.cc:654] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:46.403965 31716 tablet_bootstrap.cc:654] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:46.404565 31715 log.cc:824] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:46.405224 31716 log.cc:824] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:46.405885 31717 tablet_bootstrap.cc:654] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:46.407094 31717 log.cc:824] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:46.407814 31715 tablet_bootstrap.cc:492] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: No bootstrap required, opened a new log
I20260924 19:17:46.408115 31715 ts_tablet_manager.cc:1397] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 19:17:46.408820 31716 tablet_bootstrap.cc:492] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: No bootstrap required, opened a new log
I20260924 19:17:46.409081 31716 ts_tablet_manager.cc:1397] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 19:17:46.410136 31717 tablet_bootstrap.cc:492] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: No bootstrap required, opened a new log
I20260924 19:17:46.410394 31717 ts_tablet_manager.cc:1397] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260924 19:17:46.416627 31716 raft_consensus.cc:348] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.417083 31716 raft_consensus.cc:374] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:46.417233 31716 raft_consensus.cc:729] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2bba213d2f47d0aefd4aebff698c6b, State: Initialized, Role: FOLLOWER
I20260924 19:17:46.417927 31716 consensus_queue.cc:260] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.418929 31715 raft_consensus.cc:348] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.419597 31715 raft_consensus.cc:374] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:46.419731 31715 raft_consensus.cc:729] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec78a7b6a25144939d761361d81c2b62, State: Initialized, Role: FOLLOWER
I20260924 19:17:46.419894 31716 ts_tablet_manager.cc:1428] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260924 19:17:46.420459 31567 heartbeater.cc:503] Master 127.11.18.126:44929 was elected leader, sending a full tablet report...
I20260924 19:17:46.420457 31715 consensus_queue.cc:260] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.420899 31717 raft_consensus.cc:348] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.421423 31717 raft_consensus.cc:374] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:46.421571 31717 raft_consensus.cc:729] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472fde66e6c44ce2861c402ae7230b87, State: Initialized, Role: FOLLOWER
I20260924 19:17:46.421634 31697 heartbeater.cc:503] Master 127.11.18.126:44929 was elected leader, sending a full tablet report...
I20260924 19:17:46.422178 31717 consensus_queue.cc:260] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.423590 31715 ts_tablet_manager.cc:1428] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 19:17:46.426944 31717 ts_tablet_manager.cc:1428] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260924 19:17:46.556335 31698 tablet.cc:2369] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:17:46.565698 31568 tablet.cc:2369] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:17:46.585769 31437 tablet.cc:2369] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:17:46.647130 31722 raft_consensus.cc:482] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:46.647423 31722 raft_consensus.cc:504] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.649201 31722 leader_election.cc:288] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c2bba213d2f47d0aefd4aebff698c6b (127.11.18.66:36791), 472fde66e6c44ce2861c402ae7230b87 (127.11.18.65:46171)
I20260924 19:17:46.649230 31723 raft_consensus.cc:482] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:17:46.649544 31723 raft_consensus.cc:504] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.652113 31723 leader_election.cc:288] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c2bba213d2f47d0aefd4aebff698c6b (127.11.18.66:36791), ec78a7b6a25144939d761361d81c2b62 (127.11.18.67:39559)
I20260924 19:17:46.660208 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.649469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:36791 (local address 127.11.18.67:51027)
0924 19:17:46.649804 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:46.649822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:46.649978 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:46.650615 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:46.650620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:46.650658 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:46.650893 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.650908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.652461 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.652466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.655317 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.655340 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.659031 (+ 3691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.659041 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.659319 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.659378 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 19:17:46.659492 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 19:17:46.659734 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1}
I20260924 19:17:46.660863 31714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.650152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:44747 (local address 127.11.18.65:46171)
0924 19:17:46.650361 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.650369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.650961 (+ 592us) server_negotiation.cc:407] Connection header received
0924 19:17:46.651229 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.651234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.651298 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.651409 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:46.652074 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.653389 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.657112 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.659134 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.659321 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.659436 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.660237 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.660454 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:17:46.661238 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.649954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.67:51027 (local address 127.11.18.66:36791)
0924 19:17:46.650142 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.650150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.650177 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:17:46.650262 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.650270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.650385 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.650502 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:46.651080 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.652139 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.656042 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.657817 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.658015 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.658132 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.660636 (+ 2504us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.660846 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 19:17:46.662281 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.649469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.65:46171 (local address 127.11.18.67:44747)
0924 19:17:46.650819 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:46.650844 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:46.650980 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:46.651424 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:46.651429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:46.651455 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:46.651879 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.651895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.653674 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.653683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.656806 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.656828 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.659663 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.659672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.659977 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.660032 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 19:17:46.661595 (+ 1563us) client_negotiation.cc:240] Negotiation successful
0924 19:17:46.661826 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":576,"thread_start_us":326,"threads_started":1}
I20260924 19:17:46.664283 31522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e8b98651f64a68a7f6858fdf4c6d21" candidate_uuid: "ec78a7b6a25144939d761361d81c2b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" is_pre_election: true
I20260924 19:17:46.665060 31522 raft_consensus.cc:2454] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec78a7b6a25144939d761361d81c2b62 in term 0.
I20260924 19:17:46.665937 31585 leader_election.cc:302] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2bba213d2f47d0aefd4aebff698c6b, ec78a7b6a25144939d761361d81c2b62; no voters:
I20260924 19:17:46.666437 31722 raft_consensus.cc:2781] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:17:46.665989 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e8b98651f64a68a7f6858fdf4c6d21" candidate_uuid: "ec78a7b6a25144939d761361d81c2b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472fde66e6c44ce2861c402ae7230b87" is_pre_election: true
I20260924 19:17:46.666690 31722 raft_consensus.cc:482] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:17:46.666797 31722 raft_consensus.cc:3037] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:46.666898 31391 raft_consensus.cc:2454] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec78a7b6a25144939d761361d81c2b62 in term 0.
I20260924 19:17:46.669893 31722 raft_consensus.cc:504] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.671118 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.653024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:59875 (local address 127.11.18.67:39559)
0924 19:17:46.653210 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.653216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.662596 (+ 9380us) server_negotiation.cc:407] Connection header received
0924 19:17:46.663118 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.663125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.663217 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.663286 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:46.664602 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.665745 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.669046 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.670049 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.670269 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.670353 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.670721 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.670866 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:17:46.671155 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e8b98651f64a68a7f6858fdf4c6d21" candidate_uuid: "ec78a7b6a25144939d761361d81c2b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472fde66e6c44ce2861c402ae7230b87"
I20260924 19:17:46.671178 31522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e8b98651f64a68a7f6858fdf4c6d21" candidate_uuid: "ec78a7b6a25144939d761361d81c2b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2bba213d2f47d0aefd4aebff698c6b"
I20260924 19:17:46.671363 31391 raft_consensus.cc:3037] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:46.671412 31522 raft_consensus.cc:3037] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:46.671805 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.655672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.18.65:59715 (local address 127.11.18.66:36791)
0924 19:17:46.661551 (+ 5879us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.661559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.663005 (+ 1446us) server_negotiation.cc:407] Connection header received
0924 19:17:46.663203 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.663209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.663337 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.663400 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:17:46.664245 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.665455 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.668794 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.670111 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.670331 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.670401 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.671227 (+ 826us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.671480 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5720,"thread_start_us":6802,"threads_started":1}
I20260924 19:17:46.671957 31726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.652443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.67:39559 (local address 127.11.18.65:59875)
0924 19:17:46.662468 (+ 10025us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:46.662496 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:46.662904 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:46.663491 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:46.663495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:46.663525 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:46.664412 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.664432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.665933 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.665939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.668867 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.668878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.670299 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.670306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.670492 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.670523 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:17:46.671388 (+ 865us) client_negotiation.cc:240] Negotiation successful
0924 19:17:46.671542 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9845,"thread_start_us":10170,"threads_started":1}
I20260924 19:17:46.672305 31722 leader_election.cc:288] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [CANDIDATE]: Term 1 election: Requested vote from peers 2c2bba213d2f47d0aefd4aebff698c6b (127.11.18.66:36791), 472fde66e6c44ce2861c402ae7230b87 (127.11.18.65:46171)
I20260924 19:17:46.672371 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.652700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.66:36791 (local address 127.11.18.65:59715)
0924 19:17:46.662881 (+ 10181us) negotiation.cc:107] Waiting for socket to connect
0924 19:17:46.662898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:17:46.663041 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:17:46.663614 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:17:46.663619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:17:46.663659 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:17:46.664027 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.664036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.665619 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.665635 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.668563 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:17:46.668580 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.670535 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.670543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:17:46.670718 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.670763 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:17:46.670848 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:17:46.670992 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10051,"spinlock_wait_cycles":8448,"thread_start_us":9389,"threads_started":1}
I20260924 19:17:46.673238 31521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e8b98651f64a68a7f6858fdf4c6d21" candidate_uuid: "472fde66e6c44ce2861c402ae7230b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" is_pre_election: true
I20260924 19:17:46.673543 31652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e8b98651f64a68a7f6858fdf4c6d21" candidate_uuid: "472fde66e6c44ce2861c402ae7230b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec78a7b6a25144939d761361d81c2b62" is_pre_election: true
I20260924 19:17:46.674139 31652 raft_consensus.cc:2379] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 472fde66e6c44ce2861c402ae7230b87 in current term 1: Already voted for candidate ec78a7b6a25144939d761361d81c2b62 in this term.
I20260924 19:17:46.674999 31391 raft_consensus.cc:2454] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec78a7b6a25144939d761361d81c2b62 in term 1.
I20260924 19:17:46.675028 31327 leader_election.cc:302] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 472fde66e6c44ce2861c402ae7230b87; no voters: 2c2bba213d2f47d0aefd4aebff698c6b, ec78a7b6a25144939d761361d81c2b62
I20260924 19:17:46.675203 31522 raft_consensus.cc:2454] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec78a7b6a25144939d761361d81c2b62 in term 1.
I20260924 19:17:46.675418 31588 leader_election.cc:302] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 472fde66e6c44ce2861c402ae7230b87, ec78a7b6a25144939d761361d81c2b62; no voters:
I20260924 19:17:46.675500 31723 raft_consensus.cc:2726] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:17:46.675776 31722 raft_consensus.cc:2781] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:46.676270 31722 raft_consensus.cc:686] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 1 LEADER]: Becoming Leader. State: Replica: ec78a7b6a25144939d761361d81c2b62, State: Running, Role: LEADER
I20260924 19:17:46.676750 31722 consensus_queue.cc:237] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } }
I20260924 19:17:46.682122 31249 catalog_manager.cc:5495] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec78a7b6a25144939d761361d81c2b62 (127.11.18.67). New cstate: current_term: 1 leader_uuid: "ec78a7b6a25144939d761361d81c2b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec78a7b6a25144939d761361d81c2b62" member_type: VOTER last_known_addr { host: "127.11.18.67" port: 39559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 } health_report { overall_health: UNKNOWN } } }
I20260924 19:17:46.750010 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.743684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.11.18.67:39559)
0924 19:17:46.743902 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.743909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.743930 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:17:46.744165 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.744170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.744262 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.744436 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:17:46.744828 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.745823 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.747460 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.747928 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.748160 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.748728 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:17:46.749275 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:17:46.749510 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.749600 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.749688 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:17:46.758685 31391 raft_consensus.cc:1261] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 1 FOLLOWER]: Refusing update from remote peer ec78a7b6a25144939d761361d81c2b62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:46.758704 31522 raft_consensus.cc:1261] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 1 FOLLOWER]: Refusing update from remote peer ec78a7b6a25144939d761361d81c2b62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:17:46.759697 31722 consensus_queue.cc:1035] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "472fde66e6c44ce2861c402ae7230b87" member_type: VOTER last_known_addr { host: "127.11.18.65" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:46.760154 31728 consensus_queue.cc:1035] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2bba213d2f47d0aefd4aebff698c6b" member_type: VOTER last_known_addr { host: "127.11.18.66" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:17:46.788489 31248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:46.788841 31248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:17:46.793942 31248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ceacbf998784f1ca7ce2fb117615dd0: Sending DeleteTablet for 3 replicas of tablet f7e8b98651f64a68a7f6858fdf4c6d21
I20260924 19:17:46.795467 31502 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e8b98651f64a68a7f6858fdf4c6d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:46 UTC) from {username='slave'} at 127.0.0.1:37110
I20260924 19:17:46.795735 31632 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e8b98651f64a68a7f6858fdf4c6d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:46 UTC) from {username='slave'} at 127.0.0.1:48184
I20260924 19:17:46.795853 31371 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e8b98651f64a68a7f6858fdf4c6d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:17:46 UTC) from {username='slave'} at 127.0.0.1:52066
I20260924 19:17:46.796792 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31309
I20260924 19:17:46.808549 31740 tablet_replica.cc:330] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: stopping tablet replica
I20260924 19:17:46.809206 31741 tablet_replica.cc:330] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: stopping tablet replica
I20260924 19:17:46.809916 31740 raft_consensus.cc:2229] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:46.809917 31741 raft_consensus.cc:2229] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:17:46.810473 31740 raft_consensus.cc:2258] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:46.810767 31741 raft_consensus.cc:2258] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:46.812951 31741 ts_tablet_manager.cc:1905] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:46.815964 31742 tablet_replica.cc:330] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: stopping tablet replica
I20260924 19:17:46.816443 31742 raft_consensus.cc:2229] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:17:46.817013 31742 raft_consensus.cc:2258] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:17:46.819628 31742 ts_tablet_manager.cc:1905] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:46.820340 31741 ts_tablet_manager.cc:1918] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:46.820533 31741 log.cc:1196] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-2/wal/wals/f7e8b98651f64a68a7f6858fdf4c6d21
I20260924 19:17:46.821101 31741 ts_tablet_manager.cc:1939] T f7e8b98651f64a68a7f6858fdf4c6d21 P ec78a7b6a25144939d761361d81c2b62: Deleting consensus metadata
I20260924 19:17:46.822467 31740 ts_tablet_manager.cc:1905] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:17:46.823031 31237 catalog_manager.cc:4841] TS ec78a7b6a25144939d761361d81c2b62 (127.11.18.67:39559): tablet f7e8b98651f64a68a7f6858fdf4c6d21 (table test-table [id=1e6b335191614538bdaa8e7004a46158]) successfully deleted
I20260924 19:17:46.827312 31742 ts_tablet_manager.cc:1918] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:46.827514 31742 log.cc:1196] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-0/wal/wals/f7e8b98651f64a68a7f6858fdf4c6d21
I20260924 19:17:46.828132 31742 ts_tablet_manager.cc:1939] T f7e8b98651f64a68a7f6858fdf4c6d21 P 472fde66e6c44ce2861c402ae7230b87: Deleting consensus metadata
I20260924 19:17:46.832526 31740 ts_tablet_manager.cc:1918] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:17:46.832692 31740 log.cc:1196] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Deleting WAL directory at /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277128821086-11337-0/minicluster-data/ts-1/wal/wals/f7e8b98651f64a68a7f6858fdf4c6d21
I20260924 19:17:46.833204 31740 ts_tablet_manager.cc:1939] T f7e8b98651f64a68a7f6858fdf4c6d21 P 2c2bba213d2f47d0aefd4aebff698c6b: Deleting consensus metadata
I20260924 19:17:46.834002 31714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.797746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.11.18.65:46171)
0924 19:17:46.797998 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.798009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.809070 (+ 11061us) server_negotiation.cc:407] Connection header received
0924 19:17:46.809298 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.809308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.809512 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.809667 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:46.810659 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.812389 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.829513 (+ 17124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.830252 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.830374 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.832130 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:46.832142 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:46.832161 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:46.832972 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:46.833039 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:46.833068 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:46.833162 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.833476 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.833618 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:17:46.835093 31234 catalog_manager.cc:4841] TS 2c2bba213d2f47d0aefd4aebff698c6b (127.11.18.66:36791): tablet f7e8b98651f64a68a7f6858fdf4c6d21 (table test-table [id=1e6b335191614538bdaa8e7004a46158]) successfully deleted
I20260924 19:17:46.835301 31431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:46.835394 31237 catalog_manager.cc:4841] TS 472fde66e6c44ce2861c402ae7230b87 (127.11.18.65:46171): tablet f7e8b98651f64a68a7f6858fdf4c6d21 (table test-table [id=1e6b335191614538bdaa8e7004a46158]) successfully deleted
I20260924 19:17:46.962152 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 31309
I20260924 19:17:46.981086 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31440
I20260924 19:17:46.988458 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:46.982040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.11.18.66:36791)
0924 19:17:46.982496 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:46.982500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:46.982514 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:17:46.982566 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:46.982570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:46.982648 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:46.982763 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:46.983178 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.983877 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.985200 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:46.985814 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:46.985947 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:46.987085 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:46.987094 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:46.987109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:46.987706 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:46.987755 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:46.987775 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:46.987901 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:46.988039 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:17:46.988151 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
I20260924 19:17:46.989095 31562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:47.104976 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 31440
I20260924 19:17:47.122996 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31570
I20260924 19:17:47.131867 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:47.124117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.11.18.67:39559)
0924 19:17:47.124348 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:47.124356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:47.124379 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:17:47.124458 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:47.124464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:47.124604 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:47.124721 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:47.125215 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:47.126371 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:47.127869 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:47.128561 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:47.128715 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:47.130099 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:47.130111 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:47.130131 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:47.130945 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:47.131007 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:47.131034 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:47.131197 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:47.131363 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:17:47.131491 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:17:47.132752 31692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:47.257659 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 31570
I20260924 19:17:47.276861 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31219
I20260924 19:17:47.278146 31280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:47.401285 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 31219
2026-09-24T19:17:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3209 ms)
[----------] 6 tests from EncryptionPolicyTest (23468 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:47 dist-test-slave-0lgr krb5kdc[31750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:47 dist-test-slave-0lgr krb5kdc[31750](info): set up 2 sockets
Sep 24 19:17:47 dist-test-slave-0lgr krb5kdc[31750](info): commencing operation
krb5kdc: starting...
W20260924 19:17:48.542327 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:48 dist-test-slave-0lgr krb5kdc[31750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:17:48.811094 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:49.158129 31763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:49.158433 31763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:49.158588 31763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:49.165997 31763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:49.166117 31763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:49.166218 31763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:49.166323 31763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:49.166406 31763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:49.176463 31763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:49.177940 31763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:49.179353 31763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:49.192376 31768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:49.192314 31769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:49.192348 31771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:49.193753 31763 server_base.cc:1034] running on GCE node
I20260924 19:17:49.194612 31763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:49.196162 31763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:49.197458 31763 hybrid_clock.cc:648] HybridClock initialized: now 1790277469197429 us; error 48 us; skew 500 ppm
I20260924 19:17:49.197852 31763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:49 dist-test-slave-0lgr krb5kdc[31750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:49.204686 31763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:17:49.206468 31763 webserver.cc:460] Webserver started at http://127.0.0.1:44395/ using document root <none> and password file <none>
I20260924 19:17:49.207190 31763 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:49.207340 31763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:49.207659 31763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:49.210930 31763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "7983afe2abdb4bd89da09c94cd3b2bfd"
format_stamp: "Formatted at 2026-09-24 19:17:49 on dist-test-slave-0lgr"
server_key: "5a60ac156672ca154d97a06c5f919adf"
server_key_iv: "403939686d9ad4cfcad5ddbe9d073564"
server_key_version: "encryptionkey@0"
I20260924 19:17:49.211740 31763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "7983afe2abdb4bd89da09c94cd3b2bfd"
format_stamp: "Formatted at 2026-09-24 19:17:49 on dist-test-slave-0lgr"
server_key: "5a60ac156672ca154d97a06c5f919adf"
server_key_iv: "403939686d9ad4cfcad5ddbe9d073564"
server_key_version: "encryptionkey@0"
I20260924 19:17:49.218700 31763 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 19:17:49.222738 31778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:49.224321 31763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:17:49.224538 31763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "7983afe2abdb4bd89da09c94cd3b2bfd"
format_stamp: "Formatted at 2026-09-24 19:17:49 on dist-test-slave-0lgr"
server_key: "5a60ac156672ca154d97a06c5f919adf"
server_key_iv: "403939686d9ad4cfcad5ddbe9d073564"
server_key_version: "encryptionkey@0"
I20260924 19:17:49.224767 31763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:49.244792 31763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:49.249536 31763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:49.249867 31763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:49.269897 31763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37467
I20260924 19:17:49.269937 31829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37467 every 8 connection(s)
I20260924 19:17:49.271322 31763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:49.275123 31830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:49.277037 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 31763
I20260924 19:17:49.277251 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:49.278112 11337 external_mini_cluster.cc:1428] Setting key 704a863f4c58e03f67bd8a4675bbb0f5
I20260924 19:17:49.288470 31830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd: Bootstrap starting.
I20260924 19:17:49.292701 31830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd: Neither blocks nor log segments found. Creating new log.
Sep 24 19:17:49 dist-test-slave-0lgr krb5kdc[31750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:17:49.294024 31830 log.cc:824] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:49.297046 31830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd: No bootstrap required, opened a new log
I20260924 19:17:49.302804 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:49.280220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39817 (local address 127.0.0.1:37467)
0924 19:17:49.281217 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:49.281229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:49.281271 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:17:49.283042 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:49.283064 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:49.283460 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:49.283799 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:49.285755 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:49.287127 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:49.289059 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:49.289829 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:49.289982 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:49.295169 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:49.295180 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:49.295196 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:49.295237 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:49.298459 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:49.299330 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:49.299349 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:49.299355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:49.299526 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:49.300002 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:49.300007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:49.300011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:49.300561 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:49.300885 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:49.301441 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 19:17:49.301775 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":423,"threads_started":1}
I20260924 19:17:49.305375 31830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7983afe2abdb4bd89da09c94cd3b2bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260924 19:17:49.305960 31830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:49.306181 31830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7983afe2abdb4bd89da09c94cd3b2bfd, State: Initialized, Role: FOLLOWER
I20260924 19:17:49.306805 31830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7983afe2abdb4bd89da09c94cd3b2bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260924 19:17:49.307124 31830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:49.307161 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:17:49.307274 31830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:49.307453 31830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:49.310005 31830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7983afe2abdb4bd89da09c94cd3b2bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260924 19:17:49.310691 31830 leader_election.cc:302] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7983afe2abdb4bd89da09c94cd3b2bfd; no voters:
I20260924 19:17:49.311378 31830 leader_election.cc:288] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:49.311501 31839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:49.314051 31830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:49.318490 31839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [term 1 LEADER]: Becoming Leader. State: Replica: 7983afe2abdb4bd89da09c94cd3b2bfd, State: Running, Role: LEADER
I20260924 19:17:49.319422 31839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7983afe2abdb4bd89da09c94cd3b2bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260924 19:17:49.322906 31840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7983afe2abdb4bd89da09c94cd3b2bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7983afe2abdb4bd89da09c94cd3b2bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260924 19:17:49.323441 31840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:49.324221 31841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7983afe2abdb4bd89da09c94cd3b2bfd. Latest consensus state: current_term: 1 leader_uuid: "7983afe2abdb4bd89da09c94cd3b2bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7983afe2abdb4bd89da09c94cd3b2bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260924 19:17:49.324589 31841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:49.333695 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:49.312509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60129 (local address 127.0.0.1:37467)
0924 19:17:49.312738 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:49.312747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:49.313916 (+ 1169us) server_negotiation.cc:407] Connection header received
0924 19:17:49.314221 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:49.314227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:49.314302 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:49.314406 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:49.315941 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:49.317341 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:49.319010 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:49.319582 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:49.319667 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:49.322913 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:49.322922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:49.322929 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:49.322976 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:49.327701 (+ 4725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:49.329150 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:49.329159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:49.329165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:49.329373 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:49.330283 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:49.330291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:49.330296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:49.330876 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:49.331192 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:49.332862 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0924 19:17:49.333123 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:17:49.333877 31853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:49.339406 31853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:49.348390 31853 catalog_manager.cc:1347] Generated new cluster ID: 84739be6348444dd9c3084a9d7529c35
I20260924 19:17:49.348552 31853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:49.360152 31853 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:49.361527 31853 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:49.378830 31853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7983afe2abdb4bd89da09c94cd3b2bfd: Generated new TSK 0
I20260924 19:17:49.379722 31853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:49.388911 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31763
I20260924 19:17:49.390723 31825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:49.482611 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 31763
W20260924 19:17:49.502154 31748 connection.cc:535] client connection to 127.0.0.1:37467 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37467): Connection reset by peer (error 104)
2026-09-24T19:17:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:49 dist-test-slave-0lgr krb5kdc[31863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:49 dist-test-slave-0lgr krb5kdc[31863](info): set up 2 sockets
Sep 24 19:17:49 dist-test-slave-0lgr krb5kdc[31863](info): commencing operation
krb5kdc: starting...
W20260924 19:17:50.681867 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:50 dist-test-slave-0lgr krb5kdc[31863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:17:50.953505 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:51.315469 31876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:51.315799 31876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:51.316000 31876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:51.324453 31876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:51.324585 31876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:51.324697 31876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:51.324808 31876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:51.324896 31876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:51.336848 31876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:51.338438 31876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:51.339988 31876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:51.353229 31881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:51.353271 31884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:51.353300 31882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:51.354269 31876 server_base.cc:1034] running on GCE node
I20260924 19:17:51.355072 31876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:51.356582 31876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:51.357823 31876 hybrid_clock.cc:648] HybridClock initialized: now 1790277471357748 us; error 70 us; skew 500 ppm
I20260924 19:17:51.358235 31876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:51 dist-test-slave-0lgr krb5kdc[31863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:51.361583 31876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:17:51.363111 31876 webserver.cc:460] Webserver started at http://127.0.0.1:39561/ using document root <none> and password file <none>
I20260924 19:17:51.363811 31876 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:51.363962 31876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:51.364286 31876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:51.367440 31876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "0a49f6e8e45d40259b076e19444cfdf5"
format_stamp: "Formatted at 2026-09-24 19:17:51 on dist-test-slave-0lgr"
server_key: "2d7eff062c9c5d43f780bb3cafccb3b6"
server_key_iv: "9504abbde463959292aab0befef6a5f0"
server_key_version: "encryptionkey@0"
I20260924 19:17:51.368240 31876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "0a49f6e8e45d40259b076e19444cfdf5"
format_stamp: "Formatted at 2026-09-24 19:17:51 on dist-test-slave-0lgr"
server_key: "2d7eff062c9c5d43f780bb3cafccb3b6"
server_key_iv: "9504abbde463959292aab0befef6a5f0"
server_key_version: "encryptionkey@0"
I20260924 19:17:51.373687 31876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:17:51.377269 31891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:51.378985 31876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:51.379240 31876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "0a49f6e8e45d40259b076e19444cfdf5"
format_stamp: "Formatted at 2026-09-24 19:17:51 on dist-test-slave-0lgr"
server_key: "2d7eff062c9c5d43f780bb3cafccb3b6"
server_key_iv: "9504abbde463959292aab0befef6a5f0"
server_key_version: "encryptionkey@0"
I20260924 19:17:51.379462 31876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:51.396162 31876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:51.401265 31876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:51.401639 31876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:51.422590 31876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42523
I20260924 19:17:51.422628 31942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42523 every 8 connection(s)
I20260924 19:17:51.424152 31876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:51.428082 31943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:51.430908 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 31876
I20260924 19:17:51.431080 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:51.431627 11337 external_mini_cluster.cc:1428] Setting key 0754d52c06b67769ddaa911685e6999c
I20260924 19:17:51.440164 31943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5: Bootstrap starting.
I20260924 19:17:51.443436 31943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:51.444523 31943 log.cc:824] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:51.444728 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:51.433575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41543 (local address 127.0.0.1:42523)
0924 19:17:51.434397 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:51.434409 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:51.434450 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:17:51.436148 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:51.436171 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:51.436593 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:51.437009 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:51.437588 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:51.438937 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:51.440237 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:51.440951 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:51.441157 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:51.442461 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:51.442468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:51.442481 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:51.442998 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:51.443068 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:51.443096 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:51.443259 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:51.443528 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 19:17:51.443710 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":357,"threads_started":1}
I20260924 19:17:51.447618 31943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5: No bootstrap required, opened a new log
I20260924 19:17:51.448750 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:17:51.456139 31943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49f6e8e45d40259b076e19444cfdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } }
I20260924 19:17:51.456645 31943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:51.456753 31943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a49f6e8e45d40259b076e19444cfdf5, State: Initialized, Role: FOLLOWER
I20260924 19:17:51.457501 31943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49f6e8e45d40259b076e19444cfdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } }
I20260924 19:17:51.457785 31943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:51.457911 31943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:51.458081 31943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:51.460992 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:51.453522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54673 (local address 127.0.0.1:42523)
0924 19:17:51.453784 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:51.453794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:51.454228 (+ 434us) server_negotiation.cc:407] Connection header received
0924 19:17:51.454438 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:51.454443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:51.454520 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:51.454628 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:51.455275 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:51.456227 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:51.458172 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:51.458743 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:51.458864 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:51.460097 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:51.460102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:51.460106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:51.460125 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:51.460188 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:51.460205 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:51.460305 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:51.460545 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:17:51.460640 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:17:51.461232 31943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49f6e8e45d40259b076e19444cfdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } }
I20260924 19:17:51.461918 31943 leader_election.cc:302] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a49f6e8e45d40259b076e19444cfdf5; no voters:
I20260924 19:17:51.462702 31943 leader_election.cc:288] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:51.462841 31953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:51.463747 31953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [term 1 LEADER]: Becoming Leader. State: Replica: 0a49f6e8e45d40259b076e19444cfdf5, State: Running, Role: LEADER
I20260924 19:17:51.464347 31953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49f6e8e45d40259b076e19444cfdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } }
I20260924 19:17:51.465068 31943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:51.467365 31954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a49f6e8e45d40259b076e19444cfdf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49f6e8e45d40259b076e19444cfdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } } }
I20260924 19:17:51.467730 31954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:51.468705 31959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:51.468534 31955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a49f6e8e45d40259b076e19444cfdf5. Latest consensus state: current_term: 1 leader_uuid: "0a49f6e8e45d40259b076e19444cfdf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a49f6e8e45d40259b076e19444cfdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } } }
I20260924 19:17:51.470117 31955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:51.474344 31959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:51.484956 31959 catalog_manager.cc:1347] Generated new cluster ID: 5d6d9b9c1cce4088a0a731d83d936e50
I20260924 19:17:51.485109 31959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:51.515080 31959 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:51.517130 31959 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:51.528313 31959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a49f6e8e45d40259b076e19444cfdf5: Generated new TSK 0
I20260924 19:17:51.529531 31959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:51.541167 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31876
W20260924 19:17:51.542618 31938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41543
W20260924 19:17:51.543531 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:17:51.543725 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 31876
W20260924 19:17:51.561028 31861 connection.cc:535] client connection to 127.0.0.1:42523 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42523): Connection reset by peer (error 104)
2026-09-24T19:17:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2057 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:51 dist-test-slave-0lgr krb5kdc[31974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:51 dist-test-slave-0lgr krb5kdc[31974](info): set up 2 sockets
Sep 24 19:17:51 dist-test-slave-0lgr krb5kdc[31974](info): commencing operation
krb5kdc: starting...
W20260924 19:17:52.690424 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:52 dist-test-slave-0lgr krb5kdc[31974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:17:52.970803 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:53.317827 31987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:53.318225 31987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:53.318392 31987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:53.326025 31987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:53.326223 31987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:53.326337 31987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:53.326449 31987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:53.326542 31987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:53.337694 31987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37003
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:53.339236 31987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:53.340694 31987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:53.353899 31992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:53.353879 31993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:53.353928 31995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:53.354489 31987 server_base.cc:1034] running on GCE node
I20260924 19:17:53.355433 31987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:53.357056 31987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:53.358341 31987 hybrid_clock.cc:648] HybridClock initialized: now 1790277473358298 us; error 43 us; skew 500 ppm
I20260924 19:17:53.358763 31987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:53 dist-test-slave-0lgr krb5kdc[31974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:53.362257 31987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:17:53.363943 31987 webserver.cc:460] Webserver started at http://127.0.0.1:46321/ using document root <none> and password file <none>
I20260924 19:17:53.364629 31987 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:53.364734 31987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:53.364991 31987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:53.368232 31987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "4401241b4a8f4be79a8067664f7676a0"
format_stamp: "Formatted at 2026-09-24 19:17:53 on dist-test-slave-0lgr"
server_key: "77562ea7b78c9d609b34296582e46730"
server_key_iv: "35ff19e63d36f14bd131756378315be3"
server_key_version: "encryptionkey@0"
I20260924 19:17:53.369037 31987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "4401241b4a8f4be79a8067664f7676a0"
format_stamp: "Formatted at 2026-09-24 19:17:53 on dist-test-slave-0lgr"
server_key: "77562ea7b78c9d609b34296582e46730"
server_key_iv: "35ff19e63d36f14bd131756378315be3"
server_key_version: "encryptionkey@0"
I20260924 19:17:53.374898 31987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:17:53.379215 32002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:53.380867 31987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:17:53.381098 31987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "4401241b4a8f4be79a8067664f7676a0"
format_stamp: "Formatted at 2026-09-24 19:17:53 on dist-test-slave-0lgr"
server_key: "77562ea7b78c9d609b34296582e46730"
server_key_iv: "35ff19e63d36f14bd131756378315be3"
server_key_version: "encryptionkey@0"
I20260924 19:17:53.381374 31987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:53.388254 31987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:53.393847 31987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:53.394184 31987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:53.417815 31987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37003
I20260924 19:17:53.417866 32053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37003 every 8 connection(s)
I20260924 19:17:53.419543 31987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:53.423750 32054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:53.428000 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 31987
I20260924 19:17:53.428280 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:53.428769 11337 external_mini_cluster.cc:1428] Setting key 5d7c048d9da6b74ab11e034fa8ce4d1a
I20260924 19:17:53.436918 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:53.430658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35625 (local address 127.0.0.1:37003)
0924 19:17:53.431461 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:53.431473 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:53.431515 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:17:53.433223 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:53.433249 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:53.433719 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:53.434098 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:53.435233 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:53.435243 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:53.435259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:53.435282 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:53.435350 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:53.435371 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:53.435571 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:53.435712 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 19:17:53.435947 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":359,"threads_started":1}
I20260924 19:17:53.437883 32054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0: Bootstrap starting.
I20260924 19:17:53.441424 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:17:53.441538 32054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:53.442577 32054 log.cc:824] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:53.446314 32054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0: No bootstrap required, opened a new log
I20260924 19:17:53.449049 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:53.446378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38275 (local address 127.0.0.1:37003)
0924 19:17:53.446619 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:53.446625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:53.447127 (+ 502us) server_negotiation.cc:407] Connection header received
0924 19:17:53.447252 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:53.447256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:53.447306 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:53.447400 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:53.448354 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:53.448358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:53.448361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:53.448375 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:53.448417 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:53.448432 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:53.448530 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:53.448654 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:17:53.448706 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:17:53.455466 32054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4401241b4a8f4be79a8067664f7676a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37003 } }
I20260924 19:17:53.455893 32054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:53.456012 32054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4401241b4a8f4be79a8067664f7676a0, State: Initialized, Role: FOLLOWER
I20260924 19:17:53.456719 32054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4401241b4a8f4be79a8067664f7676a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37003 } }
I20260924 19:17:53.457029 32054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:53.457185 32054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:53.457413 32054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:53.460242 32054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4401241b4a8f4be79a8067664f7676a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37003 } }
I20260924 19:17:53.460844 32054 leader_election.cc:302] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4401241b4a8f4be79a8067664f7676a0; no voters:
I20260924 19:17:53.461714 32054 leader_election.cc:288] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:53.461902 32064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:53.462522 32064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [term 1 LEADER]: Becoming Leader. State: Replica: 4401241b4a8f4be79a8067664f7676a0, State: Running, Role: LEADER
I20260924 19:17:53.463121 32064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4401241b4a8f4be79a8067664f7676a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37003 } }
I20260924 19:17:53.465108 32064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4401241b4a8f4be79a8067664f7676a0. Latest consensus state: current_term: 1 leader_uuid: "4401241b4a8f4be79a8067664f7676a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4401241b4a8f4be79a8067664f7676a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37003 } } }
I20260924 19:17:53.465423 32064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:53.467454 32066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4401241b4a8f4be79a8067664f7676a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4401241b4a8f4be79a8067664f7676a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37003 } } }
I20260924 19:17:53.468276 32066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:53.470118 32054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:53.470647 32068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:53.477434 32068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:53.487824 32068 catalog_manager.cc:1347] Generated new cluster ID: c0b2ae71e8bc4409b742eaa368d0a0c4
I20260924 19:17:53.487977 32068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:53.513442 32068 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:53.514036 32068 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:53.530704 32068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4401241b4a8f4be79a8067664f7676a0: Generated new TSK 0
I20260924 19:17:53.531680 32068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:53.546847 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31987
W20260924 19:17:53.548180 32049 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35625
W20260924 19:17:53.548960 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 31987: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:17:53.549242 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 31987
2026-09-24T19:17:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:53 dist-test-slave-0lgr krb5kdc[32085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:53 dist-test-slave-0lgr krb5kdc[32085](info): set up 2 sockets
Sep 24 19:17:53 dist-test-slave-0lgr krb5kdc[32085](info): commencing operation
krb5kdc: starting...
W20260924 19:17:54.677983 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:54 dist-test-slave-0lgr krb5kdc[32085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:17:54.952911 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:55.318421 32098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:55.318847 32098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:55.319058 32098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:55.326956 32098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:55.327080 32098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:55.327191 32098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:55.327302 32098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:55.327381 32098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:55.337795 32098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:55.339283 32098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:55.340641 32098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:55.353276 32106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:55.353361 32104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:55.353525 32103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:55.354688 32098 server_base.cc:1034] running on GCE node
I20260924 19:17:55.355542 32098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:55.357023 32098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:55.358294 32098 hybrid_clock.cc:648] HybridClock initialized: now 1790277475358220 us; error 64 us; skew 500 ppm
I20260924 19:17:55.358696 32098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:55 dist-test-slave-0lgr krb5kdc[32085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:55.366318 32098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:17:55.368069 32098 webserver.cc:460] Webserver started at http://127.0.0.1:43617/ using document root <none> and password file <none>
I20260924 19:17:55.368803 32098 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:55.368961 32098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:55.369267 32098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:55.372406 32098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "f5f8588063a74048b62a7648f647248b"
format_stamp: "Formatted at 2026-09-24 19:17:55 on dist-test-slave-0lgr"
server_key: "0ad6e054e31bfa39a912697d0de94a51"
server_key_iv: "531949811c0f647aac695d1dba146ea0"
server_key_version: "encryptionkey@0"
I20260924 19:17:55.373238 32098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "f5f8588063a74048b62a7648f647248b"
format_stamp: "Formatted at 2026-09-24 19:17:55 on dist-test-slave-0lgr"
server_key: "0ad6e054e31bfa39a912697d0de94a51"
server_key_iv: "531949811c0f647aac695d1dba146ea0"
server_key_version: "encryptionkey@0"
I20260924 19:17:55.378793 32098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:17:55.382568 32113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:55.384063 32098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:17:55.384215 32098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "f5f8588063a74048b62a7648f647248b"
format_stamp: "Formatted at 2026-09-24 19:17:55 on dist-test-slave-0lgr"
server_key: "0ad6e054e31bfa39a912697d0de94a51"
server_key_iv: "531949811c0f647aac695d1dba146ea0"
server_key_version: "encryptionkey@0"
I20260924 19:17:55.384387 32098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:55.412228 32098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:55.417137 32098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:55.417542 32098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:55.438877 32098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39299
I20260924 19:17:55.438933 32164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39299 every 8 connection(s)
I20260924 19:17:55.440351 32098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:55.441516 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 32098
I20260924 19:17:55.441707 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:55.442179 11337 external_mini_cluster.cc:1428] Setting key 20fcca7ec931d0138338435727c3607b
I20260924 19:17:55.445227 32165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:17:55 dist-test-slave-0lgr krb5kdc[32085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:17:55.460213 32165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b: Bootstrap starting.
I20260924 19:17:55.464917 32165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:55.466455 32165 log.cc:824] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:55.469585 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:55.444131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43853 (local address 127.0.0.1:39299)
0924 19:17:55.445037 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:55.445049 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:55.445093 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:17:55.446811 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:55.446834 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:55.447227 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:55.447568 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:55.449604 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:55.451040 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:55.453108 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:55.453927 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:55.454069 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:55.459451 (+ 5382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:55.459459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:55.459476 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:55.459522 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:55.463939 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:55.464828 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:55.464839 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:55.464848 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:55.465096 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:55.465581 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:55.465586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:55.465590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:55.466406 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:55.466721 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:55.467437 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 19:17:55.467960 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":437,"threads_started":1}
I20260924 19:17:55.470512 32165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b: No bootstrap required, opened a new log
I20260924 19:17:55.474020 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:17:55.478662 32165 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f8588063a74048b62a7648f647248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260924 19:17:55.479194 32165 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:55.479321 32165 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f8588063a74048b62a7648f647248b, State: Initialized, Role: FOLLOWER
I20260924 19:17:55.480038 32165 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f8588063a74048b62a7648f647248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260924 19:17:55.480368 32165 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:55.480541 32165 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:55.480734 32165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:55.483686 32165 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f8588063a74048b62a7648f647248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260924 19:17:55.484335 32165 leader_election.cc:302] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5f8588063a74048b62a7648f647248b; no voters:
I20260924 19:17:55.485031 32165 leader_election.cc:288] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:55.485260 32175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:55.486541 32175 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [term 1 LEADER]: Becoming Leader. State: Replica: f5f8588063a74048b62a7648f647248b, State: Running, Role: LEADER
I20260924 19:17:55.487133 32175 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f8588063a74048b62a7648f647248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260924 19:17:55.487685 32165 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:55.490280 32176 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5f8588063a74048b62a7648f647248b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f8588063a74048b62a7648f647248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } } }
I20260924 19:17:55.491080 32176 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:55.490931 32177 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5f8588063a74048b62a7648f647248b. Latest consensus state: current_term: 1 leader_uuid: "f5f8588063a74048b62a7648f647248b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f8588063a74048b62a7648f647248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } } }
I20260924 19:17:55.491603 32177 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:55.501828 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:55.479025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51081 (local address 127.0.0.1:39299)
0924 19:17:55.479300 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:55.479316 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:55.479791 (+ 475us) server_negotiation.cc:407] Connection header received
0924 19:17:55.480056 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:55.480060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:55.480127 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:55.480227 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:55.482298 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:55.483206 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:55.485084 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:55.485985 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:55.486805 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:55.489191 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:55.489198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:55.489203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:17:55.489242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:55.496097 (+ 6855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:55.497914 (+ 1817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:55.497923 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:55.497931 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:55.498250 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:17:55.498937 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:17:55.498946 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:17:55.498952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:17:55.499546 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:55.500906 (+ 1360us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:55.501086 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 19:17:55.501297 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:17:55.502771 32180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:55.510169 32180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:55.521183 32180 catalog_manager.cc:1347] Generated new cluster ID: 939a84343e0643c2a0db349752461937
I20260924 19:17:55.521365 32180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:55.542945 32180 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:55.544930 32180 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:55.562749 32180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5f8588063a74048b62a7648f647248b: Generated new TSK 0
I20260924 19:17:55.563854 32180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:55.570497 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32098
I20260924 19:17:55.571830 32160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:17:55.670441 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 32098
2026-09-24T19:17:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2124 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:55 dist-test-slave-0lgr krb5kdc[32197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:55 dist-test-slave-0lgr krb5kdc[32197](info): set up 2 sockets
Sep 24 19:17:55 dist-test-slave-0lgr krb5kdc[32197](info): commencing operation
krb5kdc: starting...
W20260924 19:17:56.846354 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:57 dist-test-slave-0lgr krb5kdc[32197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:17:57.122150 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:57.479318 32210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:57.479749 32210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:57.479907 32210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:57.487864 32210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:57.487993 32210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:57.488042 32210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:57.488097 32210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:57.488174 32210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:57.498721 32210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:57.500155 32210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:57.501598 32210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:57.514349 32215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:57.514325 32216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:57.514490 32218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:57.515563 32210 server_base.cc:1034] running on GCE node
I20260924 19:17:57.516510 32210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:57.518282 32210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:57.519562 32210 hybrid_clock.cc:648] HybridClock initialized: now 1790277477519525 us; error 59 us; skew 500 ppm
I20260924 19:17:57.520035 32210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:57 dist-test-slave-0lgr krb5kdc[32197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:57.523558 32210 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:17:57.525179 32210 webserver.cc:460] Webserver started at http://127.0.0.1:41661/ using document root <none> and password file <none>
I20260924 19:17:57.525996 32210 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:57.526156 32210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:57.526492 32210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:57.529858 32210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "7b6d0af7df2f4667a13d0ae279e0dc88"
format_stamp: "Formatted at 2026-09-24 19:17:57 on dist-test-slave-0lgr"
server_key: "908aa7d8451fc7896ca487e32ab26a9d"
server_key_iv: "dd5d5bcd0491d8000202debcda09c09b"
server_key_version: "encryptionkey@0"
I20260924 19:17:57.530715 32210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "7b6d0af7df2f4667a13d0ae279e0dc88"
format_stamp: "Formatted at 2026-09-24 19:17:57 on dist-test-slave-0lgr"
server_key: "908aa7d8451fc7896ca487e32ab26a9d"
server_key_iv: "dd5d5bcd0491d8000202debcda09c09b"
server_key_version: "encryptionkey@0"
I20260924 19:17:57.536662 32210 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 19:17:57.540808 32225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:57.542615 32210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:17:57.542850 32210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "7b6d0af7df2f4667a13d0ae279e0dc88"
format_stamp: "Formatted at 2026-09-24 19:17:57 on dist-test-slave-0lgr"
server_key: "908aa7d8451fc7896ca487e32ab26a9d"
server_key_iv: "dd5d5bcd0491d8000202debcda09c09b"
server_key_version: "encryptionkey@0"
I20260924 19:17:57.543085 32210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:57.561105 32210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:57.565848 32210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:57.566174 32210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:57.586920 32210 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37537
I20260924 19:17:57.586942 32276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37537 every 8 connection(s)
I20260924 19:17:57.588393 32210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:57.589196 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 32210
I20260924 19:17:57.589447 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:57.590095 11337 external_mini_cluster.cc:1428] Setting key baa08df26f35eda3468eadc9009840b7
I20260924 19:17:57.594096 32277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:57.608376 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:57.592165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33261 (local address 127.0.0.1:37537)
0924 19:17:57.594056 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:57.594082 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:57.594151 (+ 69us) server_negotiation.cc:407] Connection header received
0924 19:17:57.596758 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:57.596791 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:57.597453 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:57.598257 (+ 804us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:57.598879 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:57.600981 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:57.602548 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:57.603672 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:57.603871 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:57.605266 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:57.605281 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:57.605302 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:57.606126 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:57.606238 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:57.606276 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:57.606405 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:57.606687 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 19:17:57.606997 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1633,"thread_start_us":1387,"threads_started":1}
I20260924 19:17:57.610013 32277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88: Bootstrap starting.
I20260924 19:17:57.612759 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:17:57.613771 32277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:57.614820 32277 log.cc:824] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:57.618669 32277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88: No bootstrap required, opened a new log
I20260924 19:17:57.624801 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:57.617792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50501 (local address 127.0.0.1:37537)
0924 19:17:57.618011 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:57.618020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:57.618456 (+ 436us) server_negotiation.cc:407] Connection header received
0924 19:17:57.618720 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:57.618747 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:57.618839 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:57.618957 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:57.619570 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:57.620372 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:57.621910 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:17:57.622434 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:17:57.622604 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:17:57.623881 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:57.623886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:57.623890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:57.623910 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:57.623950 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:57.623971 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:57.624081 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:57.624357 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 19:17:57.624465 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:17:57.628301 32277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6d0af7df2f4667a13d0ae279e0dc88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37537 } }
I20260924 19:17:57.628738 32277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:57.628858 32277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6d0af7df2f4667a13d0ae279e0dc88, State: Initialized, Role: FOLLOWER
I20260924 19:17:57.629594 32277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6d0af7df2f4667a13d0ae279e0dc88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37537 } }
I20260924 19:17:57.629875 32277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:57.630002 32277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:57.630141 32277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:57.633071 32277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6d0af7df2f4667a13d0ae279e0dc88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37537 } }
I20260924 19:17:57.633693 32277 leader_election.cc:302] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b6d0af7df2f4667a13d0ae279e0dc88; no voters:
I20260924 19:17:57.634328 32277 leader_election.cc:288] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:57.634464 32287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:57.635761 32287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [term 1 LEADER]: Becoming Leader. State: Replica: 7b6d0af7df2f4667a13d0ae279e0dc88, State: Running, Role: LEADER
I20260924 19:17:57.636380 32287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6d0af7df2f4667a13d0ae279e0dc88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37537 } }
I20260924 19:17:57.636984 32277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:57.639596 32289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b6d0af7df2f4667a13d0ae279e0dc88. Latest consensus state: current_term: 1 leader_uuid: "7b6d0af7df2f4667a13d0ae279e0dc88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6d0af7df2f4667a13d0ae279e0dc88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37537 } } }
I20260924 19:17:57.640004 32289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:57.639683 32288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b6d0af7df2f4667a13d0ae279e0dc88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6d0af7df2f4667a13d0ae279e0dc88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37537 } } }
I20260924 19:17:57.640156 32288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88 [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:57.642525 32293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:57.650074 32293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:57.665061 32293 catalog_manager.cc:1347] Generated new cluster ID: f51c38d073c7465ca1e529970cdd1eb8
I20260924 19:17:57.665258 32293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:57.722285 32293 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:57.724406 32293 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:57.737267 32293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b6d0af7df2f4667a13d0ae279e0dc88: Generated new TSK 0
I20260924 19:17:57.738257 32293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:57.755072 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32210
W20260924 19:17:57.756376 32272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33261
W20260924 19:17:57.757299 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:17:57.757509 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 32210
2026-09-24T19:17:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2088 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:57 dist-test-slave-0lgr krb5kdc[32308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:57 dist-test-slave-0lgr krb5kdc[32308](info): set up 2 sockets
Sep 24 19:17:57 dist-test-slave-0lgr krb5kdc[32308](info): commencing operation
krb5kdc: starting...
W20260924 19:17:58.881958 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:17:59 dist-test-slave-0lgr krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:17:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:17:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:17:59.158100 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:17:59.495600 32321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:17:59.495923 32321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:17:59.496091 32321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:17:59.503667 32321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:17:59.503794 32321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:17:59.503842 32321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:17:59.503896 32321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:17:59.503942 32321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:17:59.513998 32321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:17:59.515300 32321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:17:59.516573 32321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:17:59.529781 32329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:59.529609 32327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:17:59.529850 32326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:17:59.531733 32321 server_base.cc:1034] running on GCE node
I20260924 19:17:59.532688 32321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:17:59.534302 32321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:17:59.535567 32321 hybrid_clock.cc:648] HybridClock initialized: now 1790277479535528 us; error 57 us; skew 500 ppm
I20260924 19:17:59.535979 32321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:17:59 dist-test-slave-0lgr krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:17:59.539460 32321 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:17:59.540987 32321 webserver.cc:460] Webserver started at http://127.0.0.1:41429/ using document root <none> and password file <none>
I20260924 19:17:59.541821 32321 fs_manager.cc:359] Metadata directory not provided
I20260924 19:17:59.541975 32321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:17:59.542301 32321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:17:59.545459 32321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "fc721e759b634775b276506f887d112e"
format_stamp: "Formatted at 2026-09-24 19:17:59 on dist-test-slave-0lgr"
server_key: "144f1840218171f9ff262a3ef8529566"
server_key_iv: "275c8f6ef4a152f95327c72f8c12365d"
server_key_version: "encryptionkey@0"
I20260924 19:17:59.546340 32321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "fc721e759b634775b276506f887d112e"
format_stamp: "Formatted at 2026-09-24 19:17:59 on dist-test-slave-0lgr"
server_key: "144f1840218171f9ff262a3ef8529566"
server_key_iv: "275c8f6ef4a152f95327c72f8c12365d"
server_key_version: "encryptionkey@0"
I20260924 19:17:59.551839 32321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:17:59.555609 32336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:17:59.557098 32321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:17:59.557415 32321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "fc721e759b634775b276506f887d112e"
format_stamp: "Formatted at 2026-09-24 19:17:59 on dist-test-slave-0lgr"
server_key: "144f1840218171f9ff262a3ef8529566"
server_key_iv: "275c8f6ef4a152f95327c72f8c12365d"
server_key_version: "encryptionkey@0"
I20260924 19:17:59.557643 32321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:17:59.563558 32321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:17:59.568087 32321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:17:59.568393 32321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:17:59.588253 32321 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36059
I20260924 19:17:59.588303 32387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36059 every 8 connection(s)
I20260924 19:17:59.589785 32321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:17:59.593721 32388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:17:59.594913 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 32321
I20260924 19:17:59.595110 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:17:59.595577 11337 external_mini_cluster.cc:1428] Setting key 3e65326a0bab5bd3d50c0014d278bf4c
I20260924 19:17:59.604628 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:59.597564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47813 (local address 127.0.0.1:36059)
0924 19:17:59.598391 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:59.598407 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:59.598461 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:17:59.600603 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:59.600634 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:59.601131 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:59.601657 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:59.602615 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:59.602623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:59.602640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:59.602656 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:59.602720 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:59.602750 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:59.602961 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:59.603096 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 19:17:59.603280 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":343,"threads_started":1}
I20260924 19:17:59.608757 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:17:59.609127 32388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e: Bootstrap starting.
I20260924 19:17:59.613296 32388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e: Neither blocks nor log segments found. Creating new log.
I20260924 19:17:59.614682 32388 log.cc:824] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:17:59.617471 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:17:59.614332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38135 (local address 127.0.0.1:36059)
0924 19:17:59.614666 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 19:17:59.614679 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:17:59.615250 (+ 571us) server_negotiation.cc:407] Connection header received
0924 19:17:59.615429 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:17:59.615433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:17:59.615486 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:17:59.615574 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:17:59.616715 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:17:59.616720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:17:59.616724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:17:59.616746 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:17:59.616820 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:17:59.616842 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:17:59.616979 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:17:59.617079 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 19:17:59.617141 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:17:59.618449 32388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e: No bootstrap required, opened a new log
I20260924 19:17:59.626662 32388 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721e759b634775b276506f887d112e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36059 } }
I20260924 19:17:59.627069 32388 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:17:59.627140 32388 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc721e759b634775b276506f887d112e, State: Initialized, Role: FOLLOWER
I20260924 19:17:59.627753 32388 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721e759b634775b276506f887d112e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36059 } }
I20260924 19:17:59.627969 32388 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:17:59.628069 32388 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:17:59.628201 32388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:17:59.631142 32388 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721e759b634775b276506f887d112e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36059 } }
I20260924 19:17:59.631716 32388 leader_election.cc:302] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc721e759b634775b276506f887d112e; no voters:
I20260924 19:17:59.632519 32388 leader_election.cc:288] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:17:59.632628 32398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:17:59.633430 32398 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [term 1 LEADER]: Becoming Leader. State: Replica: fc721e759b634775b276506f887d112e, State: Running, Role: LEADER
I20260924 19:17:59.634061 32398 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721e759b634775b276506f887d112e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36059 } }
I20260924 19:17:59.635133 32388 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:17:59.637797 32399 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc721e759b634775b276506f887d112e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721e759b634775b276506f887d112e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36059 } } }
I20260924 19:17:59.638275 32399 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:59.641443 32400 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc721e759b634775b276506f887d112e. Latest consensus state: current_term: 1 leader_uuid: "fc721e759b634775b276506f887d112e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc721e759b634775b276506f887d112e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36059 } } }
I20260924 19:17:59.641857 32400 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e [sys.catalog]: This master's current role is: LEADER
I20260924 19:17:59.642082 32406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:17:59.649448 32406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:17:59.660581 32406 catalog_manager.cc:1347] Generated new cluster ID: 3d95f06642ed4c638ee07bc6168f03e1
I20260924 19:17:59.660749 32406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:17:59.672858 32406 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:17:59.673466 32406 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:17:59.682248 32406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc721e759b634775b276506f887d112e: Generated new TSK 0
I20260924 19:17:59.683274 32406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:17:59.699430 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32321
W20260924 19:17:59.701633 32383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47813
W20260924 19:17:59.702507 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32321: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:17:59.702695 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 32321
2026-09-24T19:17:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:17:59 dist-test-slave-0lgr krb5kdc[32419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:17:59 dist-test-slave-0lgr krb5kdc[32419](info): set up 2 sockets
Sep 24 19:17:59 dist-test-slave-0lgr krb5kdc[32419](info): commencing operation
krb5kdc: starting...
W20260924 19:18:00.870136 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:01 dist-test-slave-0lgr krb5kdc[32419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:18:01.154232 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:01.511121 32432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:01.511439 32432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:01.511538 32432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:01.519876 32432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:01.520031 32432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:01.520088 32432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:01.520159 32432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:01.520201 32432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:01.531587 32432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:01.532954 32432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:01.534410 32432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:01.547870 32437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:01.547874 32440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:01.547796 32438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:01.548875 32432 server_base.cc:1034] running on GCE node
I20260924 19:18:01.549825 32432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:01.551448 32432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:01.552754 32432 hybrid_clock.cc:648] HybridClock initialized: now 1790277481552711 us; error 41 us; skew 500 ppm
I20260924 19:18:01.553189 32432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:01 dist-test-slave-0lgr krb5kdc[32419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:01.562150 32432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:18:01.563948 32432 webserver.cc:460] Webserver started at http://127.0.0.1:46797/ using document root <none> and password file <none>
I20260924 19:18:01.564743 32432 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:01.564858 32432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:01.565140 32432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:01.568696 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "472dae5e7ee041dfa25cfef7a06ff78b"
format_stamp: "Formatted at 2026-09-24 19:18:01 on dist-test-slave-0lgr"
server_key: "21989909937f26ccb1d5c2113f71d74f"
server_key_iv: "4f7044cd47ec824f914211b64f2c81a5"
server_key_version: "encryptionkey@0"
I20260924 19:18:01.569672 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "472dae5e7ee041dfa25cfef7a06ff78b"
format_stamp: "Formatted at 2026-09-24 19:18:01 on dist-test-slave-0lgr"
server_key: "21989909937f26ccb1d5c2113f71d74f"
server_key_iv: "4f7044cd47ec824f914211b64f2c81a5"
server_key_version: "encryptionkey@0"
I20260924 19:18:01.575698 32432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 19:18:01.579952 32447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:01.581784 32432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:18:01.581944 32432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "472dae5e7ee041dfa25cfef7a06ff78b"
format_stamp: "Formatted at 2026-09-24 19:18:01 on dist-test-slave-0lgr"
server_key: "21989909937f26ccb1d5c2113f71d74f"
server_key_iv: "4f7044cd47ec824f914211b64f2c81a5"
server_key_version: "encryptionkey@0"
I20260924 19:18:01.582154 32432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:01.595994 32432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:01.601140 32432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:01.601506 32432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:01.623147 32432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36449
I20260924 19:18:01.623194 32498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36449 every 8 connection(s)
I20260924 19:18:01.624616 32432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:01.628684 32499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:01.631771 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 32432
I20260924 19:18:01.631981 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:01.632465 11337 external_mini_cluster.cc:1428] Setting key 0bb2b323b9550ce69bffe83b155bfd65
I20260924 19:18:01.641234 32499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b: Bootstrap starting.
I20260924 19:18:01.644944 32499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:01.646211 32499 log.cc:824] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:18:01 dist-test-slave-0lgr krb5kdc[32419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:18:01.650019 32499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b: No bootstrap required, opened a new log
I20260924 19:18:01.659219 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:01.634385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:60411 (local address 127.0.0.1:36449)
0924 19:18:01.635184 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:01.635196 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:01.635236 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:18:01.637577 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:01.637606 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:01.638096 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:01.638592 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:01.640276 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:01.641965 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:01.643300 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:01.644189 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:01.644462 (+ 273us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:01.649230 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:01.649243 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:01.649263 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:01.649316 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:01.653838 (+ 4522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:01.654883 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:01.654892 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:01.654902 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:01.655105 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:01.655636 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:01.655642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:01.655646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:01.656415 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:01.656755 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:01.657277 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 19:18:01.657703 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":328,"threads_started":1}
I20260924 19:18:01.661715 32499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472dae5e7ee041dfa25cfef7a06ff78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } }
I20260924 19:18:01.662153 32499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:01.662314 32499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472dae5e7ee041dfa25cfef7a06ff78b, State: Initialized, Role: FOLLOWER
I20260924 19:18:01.663033 32499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472dae5e7ee041dfa25cfef7a06ff78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } }
I20260924 19:18:01.663340 32499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:01.663513 32499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:01.663743 32499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:01.664594 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:01.666505 32499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472dae5e7ee041dfa25cfef7a06ff78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } }
I20260924 19:18:01.667089 32499 leader_election.cc:302] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472dae5e7ee041dfa25cfef7a06ff78b; no voters:
I20260924 19:18:01.668085 32499 leader_election.cc:288] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:01.668344 32506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:01.670379 32506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [term 1 LEADER]: Becoming Leader. State: Replica: 472dae5e7ee041dfa25cfef7a06ff78b, State: Running, Role: LEADER
I20260924 19:18:01.671267 32499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:01.671172 32506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472dae5e7ee041dfa25cfef7a06ff78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } }
I20260924 19:18:01.674283 32507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472dae5e7ee041dfa25cfef7a06ff78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472dae5e7ee041dfa25cfef7a06ff78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } } }
I20260924 19:18:01.674839 32507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:01.678369 32509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472dae5e7ee041dfa25cfef7a06ff78b. Latest consensus state: current_term: 1 leader_uuid: "472dae5e7ee041dfa25cfef7a06ff78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472dae5e7ee041dfa25cfef7a06ff78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } } }
I20260924 19:18:01.679890 32509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:01.683789 32514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:01.690524 32514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:01.707388 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:01.672306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:45831 (local address 127.0.0.1:36449)
0924 19:18:01.672487 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:01.672496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:01.684233 (+ 11737us) server_negotiation.cc:407] Connection header received
0924 19:18:01.684466 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:01.684484 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:01.684616 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:01.684754 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:01.687044 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:01.688743 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:01.690775 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:01.691517 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:01.691628 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:01.695751 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:01.695761 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:01.695768 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:01.695840 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:01.700362 (+ 4522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:01.702299 (+ 1937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:01.702310 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:01.702318 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:01.702556 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:01.703429 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:01.703438 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:01.703445 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:01.704093 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:01.704536 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:01.706299 (+ 1763us) server_negotiation.cc:299] Negotiation successful
0924 19:18:01.706738 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:18:01.714665 32514 catalog_manager.cc:1347] Generated new cluster ID: dfb86901949e4a749e28d2766a5cb349
I20260924 19:18:01.714818 32514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:01.727783 32514 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:01.729954 32514 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:01.739945 32514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472dae5e7ee041dfa25cfef7a06ff78b: Generated new TSK 0
I20260924 19:18:01.741199 32514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:01.755957 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32432
I20260924 19:18:01.757443 32494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:01.853549 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 32432
W20260924 19:18:01.871982 32417 connection.cc:535] client connection to 127.0.0.1:36449 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36449): Connection reset by peer (error 104)
2026-09-24T19:18:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2149 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:01 dist-test-slave-0lgr krb5kdc[32531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:01 dist-test-slave-0lgr krb5kdc[32531](info): set up 2 sockets
Sep 24 19:18:01 dist-test-slave-0lgr krb5kdc[32531](info): commencing operation
krb5kdc: starting...
W20260924 19:18:03.049969 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:03 dist-test-slave-0lgr krb5kdc[32531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:18:03.330695 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:03.700330 32544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:03.700686 32544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:03.700786 32544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:03.708917 32544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:03.709064 32544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:03.709113 32544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:03.709169 32544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:03.709224 32544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:03.719811 32544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:03.721129 32544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:03.722549 32544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:03.735970 32549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:03.736032 32550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:03.735916 32552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:03.737213 32544 server_base.cc:1034] running on GCE node
I20260924 19:18:03.738170 32544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:03.739810 32544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:03.741036 32544 hybrid_clock.cc:648] HybridClock initialized: now 1790277483740977 us; error 54 us; skew 500 ppm
I20260924 19:18:03.741539 32544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:03 dist-test-slave-0lgr krb5kdc[32531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:03.745286 32544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:18:03.747074 32544 webserver.cc:460] Webserver started at http://127.0.0.1:40525/ using document root <none> and password file <none>
I20260924 19:18:03.747854 32544 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:03.748015 32544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:03.748353 32544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:03.751771 32544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2"
format_stamp: "Formatted at 2026-09-24 19:18:03 on dist-test-slave-0lgr"
server_key: "a46c8a88a654ac9167a95a769e6bc0a9"
server_key_iv: "44fb6addfe58e924767c60e89be21134"
server_key_version: "encryptionkey@0"
I20260924 19:18:03.752606 32544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2"
format_stamp: "Formatted at 2026-09-24 19:18:03 on dist-test-slave-0lgr"
server_key: "a46c8a88a654ac9167a95a769e6bc0a9"
server_key_iv: "44fb6addfe58e924767c60e89be21134"
server_key_version: "encryptionkey@0"
I20260924 19:18:03.758553 32544 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 19:18:03.762585 32559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:03.764252 32544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:18:03.764469 32544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2"
format_stamp: "Formatted at 2026-09-24 19:18:03 on dist-test-slave-0lgr"
server_key: "a46c8a88a654ac9167a95a769e6bc0a9"
server_key_iv: "44fb6addfe58e924767c60e89be21134"
server_key_version: "encryptionkey@0"
I20260924 19:18:03.764715 32544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:03.784050 32544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:03.788918 32544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:03.789271 32544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:03.810957 32544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34549
I20260924 19:18:03.810977 32610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34549 every 8 connection(s)
I20260924 19:18:03.812400 32544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:03.816247 32611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:03.818554 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 32544
I20260924 19:18:03.818765 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:03.819268 11337 external_mini_cluster.cc:1428] Setting key 8e46a0a28c7e86bb4d83705cb441ea83
I20260924 19:18:03.829705 32611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2: Bootstrap starting.
I20260924 19:18:03.833891 32611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:03.835007 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:03.821207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:49909 (local address 127.0.0.1:34549)
0924 19:18:03.822335 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:03.822348 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:03.822392 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:18:03.824250 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:03.824276 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:03.824731 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:03.825111 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:03.825863 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:03.827462 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:03.829714 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:03.830644 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:03.830819 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:03.832312 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:03.832321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:03.832336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:03.832933 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:03.833030 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:03.833064 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:03.833198 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:03.833726 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 19:18:03.833936 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":544,"threads_started":1}
I20260924 19:18:03.835110 32611 log.cc:824] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:03.838676 32611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2: No bootstrap required, opened a new log
I20260924 19:18:03.839170 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:03.849522 32611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } }
I20260924 19:18:03.850142 32611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:03.850342 32611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca4a193d993473e8e24d1e5cbd6c8d2, State: Initialized, Role: FOLLOWER
I20260924 19:18:03.851111 32611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } }
I20260924 19:18:03.851419 32611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:03.851617 32611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:03.851913 32611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:03.852136 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:03.843872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:46249 (local address 127.0.0.1:34549)
0924 19:18:03.844068 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:03.844074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:03.844397 (+ 323us) server_negotiation.cc:407] Connection header received
0924 19:18:03.844573 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:03.844577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:03.844648 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:03.844741 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:03.845478 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:03.846326 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:03.848633 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:03.849249 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:03.849607 (+ 358us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:03.851069 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:03.851076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:03.851080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:03.851107 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:03.851187 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:03.851207 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:03.851358 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:03.851652 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 19:18:03.851719 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:18:03.855101 32611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } }
I20260924 19:18:03.855703 32611 leader_election.cc:302] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ca4a193d993473e8e24d1e5cbd6c8d2; no voters:
I20260924 19:18:03.856371 32611 leader_election.cc:288] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:03.856591 32621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:03.857486 32621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1ca4a193d993473e8e24d1e5cbd6c8d2, State: Running, Role: LEADER
I20260924 19:18:03.858104 32621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } }
I20260924 19:18:03.858808 32611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:03.861163 32623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ca4a193d993473e8e24d1e5cbd6c8d2. Latest consensus state: current_term: 1 leader_uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } } }
I20260924 19:18:03.861672 32623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:03.866696 32622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca4a193d993473e8e24d1e5cbd6c8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } } }
I20260924 19:18:03.867116 32622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2 [sys.catalog]: This master's current role is: LEADER
W20260924 19:18:03.875267 32637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:18:03.875403 32637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:18:03.875564 32629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:03.877873 32629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:03.887626 32629 catalog_manager.cc:1347] Generated new cluster ID: 037851d6d67544988a669d3e149e8959
I20260924 19:18:03.887763 32629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:03.907635 32629 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:03.909138 32629 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:03.925074 32629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ca4a193d993473e8e24d1e5cbd6c8d2: Generated new TSK 0
I20260924 19:18:03.926081 32629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:03.943203 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32544
W20260924 19:18:03.945487 32606 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.160:49909
W20260924 19:18:03.946480 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32544: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:18:03.946659 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 32544
W20260924 19:18:03.962122 32529 connection.cc:535] client connection to 127.0.0.1:34549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34549): Connection reset by peer (error 104)
2026-09-24T19:18:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2090 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:04 dist-test-slave-0lgr krb5kdc[32642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:04 dist-test-slave-0lgr krb5kdc[32642](info): set up 2 sockets
Sep 24 19:18:04 dist-test-slave-0lgr krb5kdc[32642](info): commencing operation
krb5kdc: starting...
W20260924 19:18:05.114528 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:05 dist-test-slave-0lgr krb5kdc[32642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:18:05.406484 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:05.767083 32655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:05.767424 32655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:05.767517 32655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:05.775035 32655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:05.775174 32655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:05.775349 32655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:05.775485 32655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:05.775550 32655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:05.785939 32655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:05.787387 32655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:05.789029 32655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:05.802909 32660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:05.802881 32661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:05.803158 32663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:05.803879 32655 server_base.cc:1034] running on GCE node
I20260924 19:18:05.804646 32655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:05.806362 32655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:05.807636 32655 hybrid_clock.cc:648] HybridClock initialized: now 1790277485807579 us; error 61 us; skew 500 ppm
I20260924 19:18:05.808032 32655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:05 dist-test-slave-0lgr krb5kdc[32642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:05.811668 32655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:18:05.813428 32655 webserver.cc:460] Webserver started at http://127.0.0.1:42259/ using document root <none> and password file <none>
I20260924 19:18:05.814169 32655 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:05.814276 32655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:05.814589 32655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:05.817906 32655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "b75340565069477da77affaf34704cc7"
format_stamp: "Formatted at 2026-09-24 19:18:05 on dist-test-slave-0lgr"
server_key: "dd4ebcfb1801b16abd0f6bb3bf38ed60"
server_key_iv: "1ba47403fa9647e75b6c51fe320545f3"
server_key_version: "encryptionkey@0"
I20260924 19:18:05.818753 32655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "b75340565069477da77affaf34704cc7"
format_stamp: "Formatted at 2026-09-24 19:18:05 on dist-test-slave-0lgr"
server_key: "dd4ebcfb1801b16abd0f6bb3bf38ed60"
server_key_iv: "1ba47403fa9647e75b6c51fe320545f3"
server_key_version: "encryptionkey@0"
I20260924 19:18:05.824394 32655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:18:05.829002 32670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:05.830588 32655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:18:05.830866 32655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "b75340565069477da77affaf34704cc7"
format_stamp: "Formatted at 2026-09-24 19:18:05 on dist-test-slave-0lgr"
server_key: "dd4ebcfb1801b16abd0f6bb3bf38ed60"
server_key_iv: "1ba47403fa9647e75b6c51fe320545f3"
server_key_version: "encryptionkey@0"
I20260924 19:18:05.831064 32655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:05.837636 32655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:05.842454 32655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:05.842790 32655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:05.863904 32655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44595
I20260924 19:18:05.864012 32721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44595 every 8 connection(s)
I20260924 19:18:05.865476 32655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:05.869499 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:05.874970 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 32655
I20260924 19:18:05.875137 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:05.875597 11337 external_mini_cluster.cc:1428] Setting key f76496d1322b9b40972541999512c74a
I20260924 19:18:05.881117 32722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7: Bootstrap starting.
I20260924 19:18:05.884306 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:05.877714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:59159 (local address 127.0.0.1:44595)
0924 19:18:05.878600 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:05.878613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:05.878654 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:18:05.880552 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:05.880576 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:05.881061 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:05.881552 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:05.882688 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:05.882695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:05.882708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:05.882726 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:05.882787 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:05.882807 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:05.883058 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:05.883202 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:18:05.883395 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":412,"threads_started":1}
I20260924 19:18:05.884681 32722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:05.886025 32722 log.cc:824] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:05.888686 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:05.889459 32722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7: No bootstrap required, opened a new log
I20260924 19:18:05.896898 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:05.893973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:47477 (local address 127.0.0.1:44595)
0924 19:18:05.894194 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:05.894202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:05.894626 (+ 424us) server_negotiation.cc:407] Connection header received
0924 19:18:05.894777 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:05.894783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:05.894858 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:05.894970 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:05.896087 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:05.896094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:05.896100 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:05.896127 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:05.896183 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:05.896204 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:05.896346 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:05.896523 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 19:18:05.896593 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:18:05.898406 32722 raft_consensus.cc:348] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75340565069477da77affaf34704cc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260924 19:18:05.898859 32722 raft_consensus.cc:374] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:05.898981 32722 raft_consensus.cc:729] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75340565069477da77affaf34704cc7, State: Initialized, Role: FOLLOWER
I20260924 19:18:05.899571 32722 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75340565069477da77affaf34704cc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260924 19:18:05.899843 32722 raft_consensus.cc:388] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:05.900000 32722 raft_consensus.cc:482] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:05.900206 32722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:05.903079 32722 raft_consensus.cc:504] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75340565069477da77affaf34704cc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260924 19:18:05.903707 32722 leader_election.cc:302] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75340565069477da77affaf34704cc7; no voters:
I20260924 19:18:05.904444 32722 leader_election.cc:288] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:05.904634 32732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:05.905609 32732 raft_consensus.cc:686] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [term 1 LEADER]: Becoming Leader. State: Replica: b75340565069477da77affaf34704cc7, State: Running, Role: LEADER
I20260924 19:18:05.906559 32732 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75340565069477da77affaf34704cc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260924 19:18:05.907255 32722 sys_catalog.cc:564] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:05.909660 32733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75340565069477da77affaf34704cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75340565069477da77affaf34704cc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } } }
I20260924 19:18:05.910128 32733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:05.910706 32734 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75340565069477da77affaf34704cc7. Latest consensus state: current_term: 1 leader_uuid: "b75340565069477da77affaf34704cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75340565069477da77affaf34704cc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } } }
I20260924 19:18:05.911048 32734 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:05.912596 32739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:05.919611 32739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:05.929726 32739 catalog_manager.cc:1347] Generated new cluster ID: 6637f937280147a88e782cb228010924
I20260924 19:18:05.929875 32739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:05.951797 32739 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:05.952302 32739 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:05.962966 32739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b75340565069477da77affaf34704cc7: Generated new TSK 0
I20260924 19:18:05.964363 32739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:05.975325 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32655
W20260924 19:18:05.976613 32717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.160:59159
W20260924 19:18:05.977567 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:18:05.977751 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 32655
2026-09-24T19:18:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2034 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:06 dist-test-slave-0lgr krb5kdc[32753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:06 dist-test-slave-0lgr krb5kdc[32753](info): set up 2 sockets
Sep 24 19:18:06 dist-test-slave-0lgr krb5kdc[32753](info): commencing operation
krb5kdc: starting...
W20260924 19:18:07.118103 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:07 dist-test-slave-0lgr krb5kdc[32753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:18:07.402690 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:07.775655 32766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:07.776414 32766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:07.776619 32766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:07.784286 32766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:07.784488 32766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:07.784616 32766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:07.784734 32766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:07.784826 32766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:07.794950 32766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:07.796372 32766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:07.797823 32766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:07.810456 303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:07.810391 306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:07.810541 304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:07.811518 32766 server_base.cc:1034] running on GCE node
I20260924 19:18:07.812379 32766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:07.814221 32766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:07.815470 32766 hybrid_clock.cc:648] HybridClock initialized: now 1790277487815415 us; error 72 us; skew 500 ppm
I20260924 19:18:07.815948 32766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:07 dist-test-slave-0lgr krb5kdc[32753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:07.819670 32766 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:18:07.821520 32766 webserver.cc:460] Webserver started at http://127.0.0.1:44885/ using document root <none> and password file <none>
I20260924 19:18:07.822259 32766 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:07.822434 32766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:07.822763 32766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:07.825969 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "8c93f3b75f844f1b822d67bafccc4afd"
format_stamp: "Formatted at 2026-09-24 19:18:07 on dist-test-slave-0lgr"
server_key: "bceacde20616adc669ac4e8d5257da0a"
server_key_iv: "0fa2b3fb992fdfb3136b545b9b5fc1f0"
server_key_version: "encryptionkey@0"
I20260924 19:18:07.826826 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "8c93f3b75f844f1b822d67bafccc4afd"
format_stamp: "Formatted at 2026-09-24 19:18:07 on dist-test-slave-0lgr"
server_key: "bceacde20616adc669ac4e8d5257da0a"
server_key_iv: "0fa2b3fb992fdfb3136b545b9b5fc1f0"
server_key_version: "encryptionkey@0"
I20260924 19:18:07.832342 32766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:18:07.836139 313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:07.837899 32766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:18:07.838132 32766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "8c93f3b75f844f1b822d67bafccc4afd"
format_stamp: "Formatted at 2026-09-24 19:18:07 on dist-test-slave-0lgr"
server_key: "bceacde20616adc669ac4e8d5257da0a"
server_key_iv: "0fa2b3fb992fdfb3136b545b9b5fc1f0"
server_key_version: "encryptionkey@0"
I20260924 19:18:07.838366 32766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:07.851608 32766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:07.856252 32766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:07.856586 32766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:07.876703 32766 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44763
I20260924 19:18:07.876746 364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44763 every 8 connection(s)
I20260924 19:18:07.878275 32766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:07.880385 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 32766
I20260924 19:18:07.880560 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:07.881019 11337 external_mini_cluster.cc:1428] Setting key 96c0e7c82c3c87ec438664a7787df020
I20260924 19:18:07.883606 365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:18:07 dist-test-slave-0lgr krb5kdc[32753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:18:07.898352 365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd: Bootstrap starting.
I20260924 19:18:07.901961 365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:07.903174 365 log.cc:824] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:07.906915 365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd: No bootstrap required, opened a new log
I20260924 19:18:07.907907 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:07.883189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:40645 (local address 127.0.0.1:44763)
0924 19:18:07.884112 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:07.884128 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:07.884183 (+ 55us) server_negotiation.cc:407] Connection header received
0924 19:18:07.886743 (+ 2560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:07.886773 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:07.887349 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:07.887859 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:07.889305 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:07.891730 (+ 2425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:07.893127 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:07.894210 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:07.894413 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:07.899231 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:07.899240 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:07.899257 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:07.899301 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:07.902719 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:07.903894 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:07.903906 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:07.903916 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:07.904169 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:07.904878 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:07.904882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:07.904884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:07.905503 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:07.905756 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:07.906398 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 19:18:07.906813 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":433,"threads_started":1}
I20260924 19:18:07.912166 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:07.915284 365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c93f3b75f844f1b822d67bafccc4afd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20260924 19:18:07.915828 365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:07.915959 365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c93f3b75f844f1b822d67bafccc4afd, State: Initialized, Role: FOLLOWER
I20260924 19:18:07.916661 365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c93f3b75f844f1b822d67bafccc4afd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20260924 19:18:07.916927 365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:07.917153 365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:07.917408 365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:07.920912 365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c93f3b75f844f1b822d67bafccc4afd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20260924 19:18:07.921597 365 leader_election.cc:302] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c93f3b75f844f1b822d67bafccc4afd; no voters:
I20260924 19:18:07.922488 365 leader_election.cc:288] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:07.922574 375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:07.923336 375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [term 1 LEADER]: Becoming Leader. State: Replica: 8c93f3b75f844f1b822d67bafccc4afd, State: Running, Role: LEADER
I20260924 19:18:07.924117 375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c93f3b75f844f1b822d67bafccc4afd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20260924 19:18:07.927521 365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:07.935385 377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c93f3b75f844f1b822d67bafccc4afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c93f3b75f844f1b822d67bafccc4afd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } } }
I20260924 19:18:07.936445 377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:07.936070 376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c93f3b75f844f1b822d67bafccc4afd. Latest consensus state: current_term: 1 leader_uuid: "8c93f3b75f844f1b822d67bafccc4afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c93f3b75f844f1b822d67bafccc4afd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } } }
I20260924 19:18:07.937034 376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:07.939533 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:07.917477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:37675 (local address 127.0.0.1:44763)
0924 19:18:07.917693 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:07.917702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:07.918326 (+ 624us) server_negotiation.cc:407] Connection header received
0924 19:18:07.918417 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:07.918421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:07.918490 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:07.918592 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:07.920469 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:07.921384 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:07.923587 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:07.924566 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:07.924709 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:07.928871 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:07.928882 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:07.928890 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:07.928952 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:07.934149 (+ 5197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:07.935488 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:07.935493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:07.935496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:07.935773 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:07.936434 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:07.936440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:07.936445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:07.936886 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:07.938538 (+ 1652us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:07.938779 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 19:18:07.939031 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:18:07.952756 382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:07.958873 382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:07.968657 382 catalog_manager.cc:1347] Generated new cluster ID: ad3d8677c46f4415aa8c2932c6f8f986
I20260924 19:18:07.968794 382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:07.987035 382 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:07.988981 382 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:08.000926 382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c93f3b75f844f1b822d67bafccc4afd: Generated new TSK 0
I20260924 19:18:08.002131 382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:08.008466 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 32766
I20260924 19:18:08.009809 360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:08.108915 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 32766
W20260924 19:18:08.123904 32751 connection.cc:535] client connection to 127.0.0.1:44763 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44763): Connection reset by peer (error 104)
2026-09-24T19:18:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:08 dist-test-slave-0lgr krb5kdc[398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:08 dist-test-slave-0lgr krb5kdc[398](info): set up 2 sockets
Sep 24 19:18:08 dist-test-slave-0lgr krb5kdc[398](info): commencing operation
krb5kdc: starting...
W20260924 19:18:09.249980 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:09 dist-test-slave-0lgr krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:18:09.521862 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:09.872005 411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:09.872328 411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:09.872426 411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:09.880062 411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:09.880194 411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:09.880249 411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:09.880306 411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:09.880352 411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:09.890599 411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:09.891884 411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:09.893178 411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:09.905872 419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:09.906203 417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:09.906406 416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:09.907650 411 server_base.cc:1034] running on GCE node
I20260924 19:18:09.908634 411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:09.910400 411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:09.911710 411 hybrid_clock.cc:648] HybridClock initialized: now 1790277489911654 us; error 46 us; skew 500 ppm
I20260924 19:18:09.912245 411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:09 dist-test-slave-0lgr krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:09.915987 411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:18:09.917853 411 webserver.cc:460] Webserver started at http://127.0.0.1:34567/ using document root <none> and password file <none>
I20260924 19:18:09.918592 411 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:09.918767 411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:09.919087 411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:09.922302 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "f750582d9c714a27bc1361077a9f679f"
format_stamp: "Formatted at 2026-09-24 19:18:09 on dist-test-slave-0lgr"
server_key: "1ffba4e6e4378a0cbe159313576755d0"
server_key_iv: "b77c406056a1c4682a92673963246b59"
server_key_version: "encryptionkey@0"
I20260924 19:18:09.923117 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "f750582d9c714a27bc1361077a9f679f"
format_stamp: "Formatted at 2026-09-24 19:18:09 on dist-test-slave-0lgr"
server_key: "1ffba4e6e4378a0cbe159313576755d0"
server_key_iv: "b77c406056a1c4682a92673963246b59"
server_key_version: "encryptionkey@0"
I20260924 19:18:09.929243 411 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:18:09.932986 426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:09.934813 411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:18:09.935070 411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "f750582d9c714a27bc1361077a9f679f"
format_stamp: "Formatted at 2026-09-24 19:18:09 on dist-test-slave-0lgr"
server_key: "1ffba4e6e4378a0cbe159313576755d0"
server_key_iv: "b77c406056a1c4682a92673963246b59"
server_key_version: "encryptionkey@0"
I20260924 19:18:09.935321 411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:09.957351 411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:09.962334 411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:09.962777 411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:09.983116 411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45419
I20260924 19:18:09.983135 477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45419 every 8 connection(s)
I20260924 19:18:09.984602 411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:09.988552 478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:09.990101 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 411
I20260924 19:18:09.990283 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:09.990825 11337 external_mini_cluster.cc:1428] Setting key 35d18eccce1da026943fb9397d4d7ffa
I20260924 19:18:10.004612 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f: Bootstrap starting.
I20260924 19:18:10.006224 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:09.992893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:41087 (local address 127.0.0.1:45419)
0924 19:18:09.993806 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:09.993821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:09.993866 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:18:09.995499 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:09.995524 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:09.995978 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:09.996360 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:09.997149 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:09.998562 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:10.000589 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:10.001784 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:10.001940 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:10.003545 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:10.003554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:10.003570 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:10.004136 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:10.004224 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:10.004297 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:10.004433 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:10.004973 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 19:18:10.005122 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":465,"threads_started":1}
I20260924 19:18:10.008508 478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:10.010207 478 log.cc:824] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:10.010535 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:10.014107 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f: No bootstrap required, opened a new log
I20260924 19:18:10.023567 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:10.016251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:44849 (local address 127.0.0.1:45419)
0924 19:18:10.016466 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:10.016473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:10.016895 (+ 422us) server_negotiation.cc:407] Connection header received
0924 19:18:10.017164 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:10.017170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:10.017257 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:10.017401 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:10.017966 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:10.019139 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:10.020549 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:10.021269 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:10.021446 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:10.022651 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:10.022658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:10.022663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:10.022692 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:10.022764 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:10.022787 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:10.022905 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:10.023178 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 19:18:10.023232 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:18:10.024261 478 raft_consensus.cc:348] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f750582d9c714a27bc1361077a9f679f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } }
I20260924 19:18:10.024686 478 raft_consensus.cc:374] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:10.024824 478 raft_consensus.cc:729] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f750582d9c714a27bc1361077a9f679f, State: Initialized, Role: FOLLOWER
I20260924 19:18:10.025584 478 consensus_queue.cc:260] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f750582d9c714a27bc1361077a9f679f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } }
I20260924 19:18:10.025854 478 raft_consensus.cc:388] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:10.025995 478 raft_consensus.cc:482] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:10.026185 478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:10.028625 478 raft_consensus.cc:504] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f750582d9c714a27bc1361077a9f679f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } }
I20260924 19:18:10.029268 478 leader_election.cc:302] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f750582d9c714a27bc1361077a9f679f; no voters:
I20260924 19:18:10.030011 478 leader_election.cc:288] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:10.030180 488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:10.030894 488 raft_consensus.cc:686] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [term 1 LEADER]: Becoming Leader. State: Replica: f750582d9c714a27bc1361077a9f679f, State: Running, Role: LEADER
I20260924 19:18:10.031471 488 consensus_queue.cc:237] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f750582d9c714a27bc1361077a9f679f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } }
I20260924 19:18:10.032271 478 sys_catalog.cc:564] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:10.034792 490 sys_catalog.cc:455] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f750582d9c714a27bc1361077a9f679f. Latest consensus state: current_term: 1 leader_uuid: "f750582d9c714a27bc1361077a9f679f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f750582d9c714a27bc1361077a9f679f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } } }
I20260924 19:18:10.035434 490 sys_catalog.cc:458] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:10.034806 489 sys_catalog.cc:455] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f750582d9c714a27bc1361077a9f679f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f750582d9c714a27bc1361077a9f679f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45419 } } }
I20260924 19:18:10.036021 489 sys_catalog.cc:458] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:10.038019 497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:10.043634 497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:10.055171 497 catalog_manager.cc:1347] Generated new cluster ID: 2e7a92ab14584583a3072fe2dbd09ec6
I20260924 19:18:10.055343 497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:10.066069 497 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:10.067565 497 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:10.074962 497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f750582d9c714a27bc1361077a9f679f: Generated new TSK 0
I20260924 19:18:10.075945 497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:10.085664 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 411
W20260924 19:18:10.086961 473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.160:41087
W20260924 19:18:10.087800 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:18:10.087934 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 411
W20260924 19:18:10.103437 396 connection.cc:535] client connection to 127.0.0.1:45419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45419): Connection reset by peer (error 104)
2026-09-24T19:18:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:10 dist-test-slave-0lgr krb5kdc[509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:10 dist-test-slave-0lgr krb5kdc[509](info): set up 2 sockets
Sep 24 19:18:10 dist-test-slave-0lgr krb5kdc[509](info): commencing operation
krb5kdc: starting...
W20260924 19:18:11.246148 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:11 dist-test-slave-0lgr krb5kdc[509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:18:11.526520 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:11.871038 522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:11.871335 522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:11.871429 522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:11.878618 522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:11.878744 522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:11.878791 522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:11.878845 522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:11.878890 522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:11.888852 522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:11.890189 522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:11.891459 522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:11.905157 527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:11.905149 530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:11.905207 528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:11.906288 522 server_base.cc:1034] running on GCE node
I20260924 19:18:11.907120 522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:11.908598 522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:11.909821 522 hybrid_clock.cc:648] HybridClock initialized: now 1790277491909751 us; error 59 us; skew 500 ppm
I20260924 19:18:11.910224 522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:11 dist-test-slave-0lgr krb5kdc[509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:11.913561 522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:18:11.915341 522 webserver.cc:460] Webserver started at http://127.0.0.1:36857/ using document root <none> and password file <none>
I20260924 19:18:11.916013 522 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:11.916126 522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:11.916374 522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:11.920050 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "e951b7aee6c7436a9de64e03194ef4e7"
format_stamp: "Formatted at 2026-09-24 19:18:11 on dist-test-slave-0lgr"
server_key: "cc4ade24146fbdeb7d32c06170040ce5"
server_key_iv: "bad86acd801c2cd86e98ccf68d77709f"
server_key_version: "encryptionkey@0"
I20260924 19:18:11.920814 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "e951b7aee6c7436a9de64e03194ef4e7"
format_stamp: "Formatted at 2026-09-24 19:18:11 on dist-test-slave-0lgr"
server_key: "cc4ade24146fbdeb7d32c06170040ce5"
server_key_iv: "bad86acd801c2cd86e98ccf68d77709f"
server_key_version: "encryptionkey@0"
I20260924 19:18:11.926601 522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:18:11.930451 537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:11.932301 522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260924 19:18:11.932613 522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "e951b7aee6c7436a9de64e03194ef4e7"
format_stamp: "Formatted at 2026-09-24 19:18:11 on dist-test-slave-0lgr"
server_key: "cc4ade24146fbdeb7d32c06170040ce5"
server_key_iv: "bad86acd801c2cd86e98ccf68d77709f"
server_key_version: "encryptionkey@0"
I20260924 19:18:11.932859 522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:11.939253 522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:11.944412 522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:11.944811 522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:11.964771 522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42535
I20260924 19:18:11.964826 588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42535 every 8 connection(s)
I20260924 19:18:11.966325 522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:11.970317 589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:11.973495 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 522
I20260924 19:18:11.973699 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:11.974303 11337 external_mini_cluster.cc:1428] Setting key e660f40e3e4597c15718ea4b5a2e26cf
I20260924 19:18:11.983325 589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7: Bootstrap starting.
I20260924 19:18:11.983596 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:11.976399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:48267 (local address 127.0.0.1:42535)
0924 19:18:11.977394 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:11.977413 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:11.977466 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:18:11.979381 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:11.979405 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:11.979810 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:11.980279 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:11.981486 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:11.981493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:11.981507 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:11.981539 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:11.981596 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:11.981624 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:11.981900 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:11.982070 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 19:18:11.982281 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":399,"threads_started":1}
I20260924 19:18:11.987361 589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:11.988312 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:11.988610 589 log.cc:824] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:11.991631 589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7: No bootstrap required, opened a new log
I20260924 19:18:11.996353 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:11.993219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:39961 (local address 127.0.0.1:42535)
0924 19:18:11.993446 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:11.993452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:11.993911 (+ 459us) server_negotiation.cc:407] Connection header received
0924 19:18:11.994061 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:11.994066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:11.994131 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:11.994262 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:11.995579 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:11.995585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:11.995591 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:11.995618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:11.995671 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:11.995698 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:11.995875 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:11.996001 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:18:11.996065 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:18:12.001025 589 raft_consensus.cc:348] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e951b7aee6c7436a9de64e03194ef4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260924 19:18:12.001535 589 raft_consensus.cc:374] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:12.001621 589 raft_consensus.cc:729] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e951b7aee6c7436a9de64e03194ef4e7, State: Initialized, Role: FOLLOWER
I20260924 19:18:12.002254 589 consensus_queue.cc:260] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e951b7aee6c7436a9de64e03194ef4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260924 19:18:12.002545 589 raft_consensus.cc:388] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:12.002648 589 raft_consensus.cc:482] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:12.002774 589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:12.005808 589 raft_consensus.cc:504] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e951b7aee6c7436a9de64e03194ef4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260924 19:18:12.006570 589 leader_election.cc:302] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e951b7aee6c7436a9de64e03194ef4e7; no voters:
I20260924 19:18:12.007341 589 leader_election.cc:288] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:12.007475 599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:12.008319 599 raft_consensus.cc:686] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [term 1 LEADER]: Becoming Leader. State: Replica: e951b7aee6c7436a9de64e03194ef4e7, State: Running, Role: LEADER
I20260924 19:18:12.009078 599 consensus_queue.cc:237] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e951b7aee6c7436a9de64e03194ef4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260924 19:18:12.009888 589 sys_catalog.cc:564] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:12.012176 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e951b7aee6c7436a9de64e03194ef4e7. Latest consensus state: current_term: 1 leader_uuid: "e951b7aee6c7436a9de64e03194ef4e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e951b7aee6c7436a9de64e03194ef4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } } }
I20260924 19:18:12.012552 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:12.013159 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e951b7aee6c7436a9de64e03194ef4e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e951b7aee6c7436a9de64e03194ef4e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } } }
I20260924 19:18:12.013479 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:12.013482 604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:12.019656 604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:12.028419 604 catalog_manager.cc:1347] Generated new cluster ID: 767360fa39894dca8002205c9d949e7e
I20260924 19:18:12.028550 604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:12.063094 604 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:12.063584 604 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:12.074739 604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e951b7aee6c7436a9de64e03194ef4e7: Generated new TSK 0
I20260924 19:18:12.075920 604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:12.092694 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 522
W20260924 19:18:12.094313 584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.160:48267
W20260924 19:18:12.095080 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 522: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:18:12.095254 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 522
2026-09-24T19:18:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2006 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:12 dist-test-slave-0lgr krb5kdc[620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:12 dist-test-slave-0lgr krb5kdc[620](info): set up 2 sockets
Sep 24 19:18:12 dist-test-slave-0lgr krb5kdc[620](info): commencing operation
krb5kdc: starting...
W20260924 19:18:13.226539 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:13 dist-test-slave-0lgr krb5kdc[620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:18:13.516943 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36127
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:36127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:13.880863 633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:13.881281 633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:13.881472 633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:13.889104 633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:13.889294 633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:13.889431 633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:13.889539 633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:13.889605 633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:13.899847 633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:36127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:36127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:13.901274 633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:13.902642 633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:13.915469 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:13.915504 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:13.915469 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:13.923743 633 server_base.cc:1034] running on GCE node
I20260924 19:18:13.924731 633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:13.926452 633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:13.927757 633 hybrid_clock.cc:648] HybridClock initialized: now 1790277493927681 us; error 65 us; skew 500 ppm
I20260924 19:18:13.928150 633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:13 dist-test-slave-0lgr krb5kdc[620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277493, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:13.938270 633 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:18:13.940213 633 webserver.cc:460] Webserver started at http://127.11.18.126:40757/ using document root <none> and password file <none>
I20260924 19:18:13.941007 633 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:13.941183 633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:13.941617 633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:13.944804 633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8"
format_stamp: "Formatted at 2026-09-24 19:18:13 on dist-test-slave-0lgr"
server_key: "f65c9a64ea18eff2b3fe810026f567da"
server_key_iv: "585617f1333e227d6a0a4f6855d6f90b"
server_key_version: "encryptionkey@0"
I20260924 19:18:13.945727 633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8"
format_stamp: "Formatted at 2026-09-24 19:18:13 on dist-test-slave-0lgr"
server_key: "f65c9a64ea18eff2b3fe810026f567da"
server_key_iv: "585617f1333e227d6a0a4f6855d6f90b"
server_key_version: "encryptionkey@0"
I20260924 19:18:13.951499 633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:18:13.955658 648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:13.957383 633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:18:13.957609 633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8"
format_stamp: "Formatted at 2026-09-24 19:18:13 on dist-test-slave-0lgr"
server_key: "f65c9a64ea18eff2b3fe810026f567da"
server_key_iv: "585617f1333e227d6a0a4f6855d6f90b"
server_key_version: "encryptionkey@0"
I20260924 19:18:13.957872 633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:13.972841 633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:13.978020 633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:13.978312 633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:14.001080 633 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:36127
I20260924 19:18:14.001102 699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:36127 every 8 connection(s)
I20260924 19:18:14.002729 633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:14.006907 700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:14.007355 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 633
I20260924 19:18:14.007567 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:14.008122 11337 external_mini_cluster.cc:1428] Setting key dc76b04ec032c5d899d4ab2a0cdf4df0
I20260924 19:18:14.023479 700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8: Bootstrap starting.
Sep 24 19:18:14 dist-test-slave-0lgr krb5kdc[620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:18:14.027576 700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:14.028829 700 log.cc:824] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:14.033080 700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8: No bootstrap required, opened a new log
I20260924 19:18:14.037002 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:14.010032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38579 (local address 127.11.18.126:36127)
0924 19:18:14.010939 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:14.010954 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:14.011006 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:18:14.014050 (+ 3044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:14.014084 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:14.014724 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:14.015417 (+ 693us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:14.016752 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:14.018963 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:14.020383 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:14.021911 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:14.022156 (+ 245us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:14.027218 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:14.027228 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:14.027244 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:14.027294 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:14.031594 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:14.032884 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:14.032891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:14.032898 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:14.033078 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:14.033659 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:14.033668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:14.033676 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:14.034303 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:14.034695 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:14.035246 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 19:18:14.035583 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":451,"threads_started":1}
I20260924 19:18:14.041781 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:14.042431 700 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36127 } }
I20260924 19:18:14.042847 700 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:14.042960 700 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cc2c47b8cd42fc92cc15c281d3dbe8, State: Initialized, Role: FOLLOWER
I20260924 19:18:14.043597 700 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36127 } }
I20260924 19:18:14.043857 700 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:14.044005 700 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:14.044193 700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:14.046746 700 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36127 } }
I20260924 19:18:14.047356 700 leader_election.cc:302] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3cc2c47b8cd42fc92cc15c281d3dbe8; no voters:
I20260924 19:18:14.048069 700 leader_election.cc:288] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:14.048254 710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:14.050884 700 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:14.053707 710 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [term 1 LEADER]: Becoming Leader. State: Replica: e3cc2c47b8cd42fc92cc15c281d3dbe8, State: Running, Role: LEADER
I20260924 19:18:14.054395 710 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36127 } }
I20260924 19:18:14.063771 712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3cc2c47b8cd42fc92cc15c281d3dbe8. Latest consensus state: current_term: 1 leader_uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36127 } } }
I20260924 19:18:14.064280 712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:14.064110 711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cc2c47b8cd42fc92cc15c281d3dbe8" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 36127 } } }
I20260924 19:18:14.064839 711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:14.066685 721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:14.072880 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:14.046985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52239 (local address 127.11.18.126:36127)
0924 19:18:14.047179 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:14.047185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:14.047817 (+ 632us) server_negotiation.cc:407] Connection header received
0924 19:18:14.048065 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:14.048070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:14.048161 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:14.048278 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:14.050781 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:14.052729 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:14.055517 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:14.056554 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:14.056762 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:14.061170 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:14.061182 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:14.061191 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:14.061258 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:14.068728 (+ 7470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:14.070155 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:14.070164 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:14.070168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:14.070351 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:14.070940 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:14.070946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:14.070950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:14.071462 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:14.071783 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:14.072248 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 19:18:14.072463 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:18:14.073783 721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:14.084869 721 catalog_manager.cc:1347] Generated new cluster ID: 37b0f2aa6e3047078a7816d655785c35
I20260924 19:18:14.085022 721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:14.092159 721 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:14.094265 721 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:14.102124 721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3cc2c47b8cd42fc92cc15c281d3dbe8: Generated new TSK 0
I20260924 19:18:14.103144 721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:14.114497 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 633
I20260924 19:18:14.116528 695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:14.214596 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 633
W20260924 19:18:14.233289 618 connection.cc:535] client connection to 127.11.18.126:36127 recv error: Network error: failed to read from TLS socket (remote: 127.11.18.126:36127): Connection reset by peer (error 104)
2026-09-24T19:18:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:14 dist-test-slave-0lgr krb5kdc[732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:14 dist-test-slave-0lgr krb5kdc[732](info): set up 2 sockets
Sep 24 19:18:14 dist-test-slave-0lgr krb5kdc[732](info): commencing operation
krb5kdc: starting...
W20260924 19:18:15.370193 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:15 dist-test-slave-0lgr krb5kdc[732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:18:15.650347 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:33625
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:15.993837 745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:15.994180 745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:15.994377 745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:16.002655 745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:16.002784 745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:16.002884 745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:16.002981 745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:16.003054 745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:16.014458 745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:33625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:16.015939 745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:16.017414 745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:16.030568 750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:16.030681 751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:16.030637 753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:16.031865 745 server_base.cc:1034] running on GCE node
I20260924 19:18:16.032846 745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:16.034552 745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:16.035805 745 hybrid_clock.cc:648] HybridClock initialized: now 1790277496035761 us; error 41 us; skew 500 ppm
I20260924 19:18:16.036306 745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:16 dist-test-slave-0lgr krb5kdc[732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277496, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:16.039820 745 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:18:16.041630 745 webserver.cc:460] Webserver started at http://127.11.18.126:34563/ using document root <none> and password file <none>
I20260924 19:18:16.042330 745 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:16.042466 745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:16.042800 745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:16.045951 745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "4cafa02b2e64413f858bf47d024cff8e"
format_stamp: "Formatted at 2026-09-24 19:18:16 on dist-test-slave-0lgr"
server_key: "71a7d5703232fc4dcb0dcce491032c42"
server_key_iv: "76c98bbafe0c0ebf2856fa99cfd7c72e"
server_key_version: "encryptionkey@0"
I20260924 19:18:16.046751 745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "4cafa02b2e64413f858bf47d024cff8e"
format_stamp: "Formatted at 2026-09-24 19:18:16 on dist-test-slave-0lgr"
server_key: "71a7d5703232fc4dcb0dcce491032c42"
server_key_iv: "76c98bbafe0c0ebf2856fa99cfd7c72e"
server_key_version: "encryptionkey@0"
I20260924 19:18:16.052424 745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:18:16.056521 760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:16.058223 745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:18:16.058461 745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "4cafa02b2e64413f858bf47d024cff8e"
format_stamp: "Formatted at 2026-09-24 19:18:16 on dist-test-slave-0lgr"
server_key: "71a7d5703232fc4dcb0dcce491032c42"
server_key_iv: "76c98bbafe0c0ebf2856fa99cfd7c72e"
server_key_version: "encryptionkey@0"
I20260924 19:18:16.058708 745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:16.074533 745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:16.079334 745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:16.079641 745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:16.100070 745 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:33625
I20260924 19:18:16.100100 811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:33625 every 8 connection(s)
I20260924 19:18:16.101949 745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:16.105906 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:16.109555 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 745
I20260924 19:18:16.109758 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:16.110311 11337 external_mini_cluster.cc:1428] Setting key 5b8dff5a1818d667e127e6cebb290668
I20260924 19:18:16.120039 812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e: Bootstrap starting.
I20260924 19:18:16.124123 812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:16.125516 812 log.cc:824] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:16.126147 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:16.112491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36139 (local address 127.11.18.126:33625)
0924 19:18:16.113458 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:16.113471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:16.113515 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:18:16.115343 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:16.115368 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:16.115842 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:16.116209 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:16.116937 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:16.118621 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:16.120834 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:16.121641 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:16.121785 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:16.123333 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:16.123342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:16.123357 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:16.123966 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:16.124075 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:16.124103 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:16.124309 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:16.124666 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 19:18:16.124879 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":449,"threads_started":1}
I20260924 19:18:16.128711 812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e: No bootstrap required, opened a new log
I20260924 19:18:16.130405 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:16.140112 812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cafa02b2e64413f858bf47d024cff8e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 33625 } }
I20260924 19:18:16.140602 812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:16.140728 812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cafa02b2e64413f858bf47d024cff8e, State: Initialized, Role: FOLLOWER
I20260924 19:18:16.141585 812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cafa02b2e64413f858bf47d024cff8e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 33625 } }
I20260924 19:18:16.141938 812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:16.142167 812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:16.142417 812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:16.144580 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:16.136369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45983 (local address 127.11.18.126:33625)
0924 19:18:16.136594 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:16.136606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:16.137042 (+ 436us) server_negotiation.cc:407] Connection header received
0924 19:18:16.137229 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:16.137235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:16.137351 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:16.137490 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:16.138073 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:16.139402 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:16.141098 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:16.141898 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:16.142066 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:16.143582 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:16.143587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:16.143591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:16.143614 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:16.143660 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:16.143677 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:16.143831 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:16.144100 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 19:18:16.144241 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:18:16.145988 812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cafa02b2e64413f858bf47d024cff8e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 33625 } }
I20260924 19:18:16.146610 812 leader_election.cc:302] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cafa02b2e64413f858bf47d024cff8e; no voters:
I20260924 19:18:16.147287 812 leader_election.cc:288] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:16.147475 822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:16.148921 822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [term 1 LEADER]: Becoming Leader. State: Replica: 4cafa02b2e64413f858bf47d024cff8e, State: Running, Role: LEADER
I20260924 19:18:16.149494 822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cafa02b2e64413f858bf47d024cff8e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 33625 } }
I20260924 19:18:16.149818 812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:16.152715 823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cafa02b2e64413f858bf47d024cff8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cafa02b2e64413f858bf47d024cff8e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 33625 } } }
I20260924 19:18:16.152762 824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cafa02b2e64413f858bf47d024cff8e. Latest consensus state: current_term: 1 leader_uuid: "4cafa02b2e64413f858bf47d024cff8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cafa02b2e64413f858bf47d024cff8e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 33625 } } }
I20260924 19:18:16.153231 824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:16.153234 823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:16.158522 830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:16.165122 830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:16.174846 830 catalog_manager.cc:1347] Generated new cluster ID: e3545c7fb41e4c8fadbbaf3c32650414
I20260924 19:18:16.174983 830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:16.187172 830 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:16.189347 830 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:16.205456 830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cafa02b2e64413f858bf47d024cff8e: Generated new TSK 0
I20260924 19:18:16.206439 830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:16.222070 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 745
W20260924 19:18:16.223898 807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36139
W20260924 19:18:16.224762 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 745: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:18:16.224931 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 745
W20260924 19:18:16.240224 730 connection.cc:535] client connection to 127.11.18.126:33625 recv error: Network error: failed to read from TLS socket (remote: 127.11.18.126:33625): Connection reset by peer (error 104)
2026-09-24T19:18:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2007 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:16 dist-test-slave-0lgr krb5kdc[843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:16 dist-test-slave-0lgr krb5kdc[843](info): set up 2 sockets
Sep 24 19:18:16 dist-test-slave-0lgr krb5kdc[843](info): commencing operation
krb5kdc: starting...
W20260924 19:18:17.362056 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:17 dist-test-slave-0lgr krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:18:17.655154 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38197
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:17.997458 856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:17.997758 856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:17.997864 856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:18.005388 856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:18.005501 856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:18.005548 856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:18.005609 856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:18.005656 856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:18.015832 856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:38197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:18.017151 856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:18.018550 856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:18.030864 864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:18.030886 862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:18.030963 861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:18.031744 856 server_base.cc:1034] running on GCE node
I20260924 19:18:18.032495 856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:18.034109 856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:18.035389 856 hybrid_clock.cc:648] HybridClock initialized: now 1790277498035347 us; error 65 us; skew 500 ppm
I20260924 19:18:18.035799 856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:18 dist-test-slave-0lgr krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277498, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:18.039240 856 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:18:18.040982 856 webserver.cc:460] Webserver started at http://127.11.18.126:41841/ using document root <none> and password file <none>
I20260924 19:18:18.041925 856 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:18.042092 856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:18.042438 856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:18.045867 856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "205e366cb3424670a36dcfc4a27d0c7a"
format_stamp: "Formatted at 2026-09-24 19:18:18 on dist-test-slave-0lgr"
server_key: "1296f4cb3d47a1ca50af09a44f112e54"
server_key_iv: "1fff4ed11dd009e4bd413342836ca231"
server_key_version: "encryptionkey@0"
I20260924 19:18:18.046751 856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "205e366cb3424670a36dcfc4a27d0c7a"
format_stamp: "Formatted at 2026-09-24 19:18:18 on dist-test-slave-0lgr"
server_key: "1296f4cb3d47a1ca50af09a44f112e54"
server_key_iv: "1fff4ed11dd009e4bd413342836ca231"
server_key_version: "encryptionkey@0"
I20260924 19:18:18.052532 856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:18:18.056407 871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:18.057940 856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:18:18.058136 856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "205e366cb3424670a36dcfc4a27d0c7a"
format_stamp: "Formatted at 2026-09-24 19:18:18 on dist-test-slave-0lgr"
server_key: "1296f4cb3d47a1ca50af09a44f112e54"
server_key_iv: "1fff4ed11dd009e4bd413342836ca231"
server_key_version: "encryptionkey@0"
I20260924 19:18:18.058357 856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:18.067832 856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:18.072865 856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:18.073215 856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:18.095201 856 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:38197
I20260924 19:18:18.095297 922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:38197 every 8 connection(s)
I20260924 19:18:18.096756 856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:18.100924 923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:18.103557 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 856
I20260924 19:18:18.103750 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:18.104266 11337 external_mini_cluster.cc:1428] Setting key 38bcdee1176d8be07a85238e653b047e
I20260924 19:18:18.113466 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:18.106289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49025 (local address 127.11.18.126:38197)
0924 19:18:18.107181 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:18.107196 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:18.107320 (+ 124us) server_negotiation.cc:407] Connection header received
0924 19:18:18.109573 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:18.109601 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:18.110001 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:18.110466 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:18.111751 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:18.111759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:18.111772 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:18.111787 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:18.111843 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:18.111863 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:18.112115 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:18.112264 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 19:18:18.112456 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":443,"threads_started":1}
I20260924 19:18:18.114028 923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a: Bootstrap starting.
I20260924 19:18:18.117764 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:18.117982 923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:18.119014 923 log.cc:824] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:18.122315 923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a: No bootstrap required, opened a new log
I20260924 19:18:18.124979 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:18.122109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60731 (local address 127.11.18.126:38197)
0924 19:18:18.122296 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:18.122308 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:18.122799 (+ 491us) server_negotiation.cc:407] Connection header received
0924 19:18:18.122921 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:18.122927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:18.122995 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:18.123115 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:18.124203 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:18.124208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:18.124213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:18.124235 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:18.124283 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:18.124309 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:18.124439 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:18.124614 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:18:18.124680 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:18:18.131727 923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205e366cb3424670a36dcfc4a27d0c7a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38197 } }
I20260924 19:18:18.132110 923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:18.132186 923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205e366cb3424670a36dcfc4a27d0c7a, State: Initialized, Role: FOLLOWER
I20260924 19:18:18.132705 923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205e366cb3424670a36dcfc4a27d0c7a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38197 } }
I20260924 19:18:18.132905 923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:18.133002 923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:18.133126 923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:18.135718 923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205e366cb3424670a36dcfc4a27d0c7a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38197 } }
I20260924 19:18:18.136188 923 leader_election.cc:302] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 205e366cb3424670a36dcfc4a27d0c7a; no voters:
I20260924 19:18:18.136736 923 leader_election.cc:288] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:18.136956 933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:18.137878 933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [term 1 LEADER]: Becoming Leader. State: Replica: 205e366cb3424670a36dcfc4a27d0c7a, State: Running, Role: LEADER
I20260924 19:18:18.138598 933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205e366cb3424670a36dcfc4a27d0c7a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38197 } }
I20260924 19:18:18.139211 923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:18.141450 935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 205e366cb3424670a36dcfc4a27d0c7a. Latest consensus state: current_term: 1 leader_uuid: "205e366cb3424670a36dcfc4a27d0c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205e366cb3424670a36dcfc4a27d0c7a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38197 } } }
I20260924 19:18:18.142016 935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:18.141536 934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "205e366cb3424670a36dcfc4a27d0c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205e366cb3424670a36dcfc4a27d0c7a" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 38197 } } }
I20260924 19:18:18.142395 934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:18.146777 941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:18.152653 941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:18.161496 941 catalog_manager.cc:1347] Generated new cluster ID: 2481e4809e3e493590c92a8ad122c25d
I20260924 19:18:18.161626 941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:18.184419 941 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:18.184953 941 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:18.194036 941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 205e366cb3424670a36dcfc4a27d0c7a: Generated new TSK 0
I20260924 19:18:18.194973 941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:18.204149 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 856
W20260924 19:18:18.205533 918 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49025
W20260924 19:18:18.206300 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:18:18.206459 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 856
2026-09-24T19:18:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:18 dist-test-slave-0lgr krb5kdc[954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:18 dist-test-slave-0lgr krb5kdc[954](info): set up 2 sockets
Sep 24 19:18:18 dist-test-slave-0lgr krb5kdc[954](info): commencing operation
krb5kdc: starting...
W20260924 19:18:19.351323 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:19 dist-test-slave-0lgr krb5kdc[954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:18:19.639599 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45587
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:19.998389 967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:19.998716 967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:19.998896 967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:20.007555 967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:20.007694 967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:20.007818 967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:20.007916 967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:20.008023 967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:20.019668 967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:20.021132 967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:20.022567 967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:20.035970 972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:20.036057 975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:20.036082 973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:20.037521 967 server_base.cc:1034] running on GCE node
I20260924 19:18:20.038389 967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:20.039983 967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:20.041241 967 hybrid_clock.cc:648] HybridClock initialized: now 1790277500041192 us; error 44 us; skew 500 ppm
I20260924 19:18:20.041690 967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:20 dist-test-slave-0lgr krb5kdc[954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277500, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:20.050730 967 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:18:20.052542 967 webserver.cc:460] Webserver started at http://127.11.18.126:40513/ using document root <none> and password file <none>
I20260924 19:18:20.053269 967 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:20.053444 967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:20.053751 967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:20.056851 967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92"
format_stamp: "Formatted at 2026-09-24 19:18:20 on dist-test-slave-0lgr"
server_key: "4e9935bb8a6444342aa1812d9aac6d3e"
server_key_iv: "1fe1c1a669afcd1fa553c8d5307b583d"
server_key_version: "encryptionkey@0"
I20260924 19:18:20.057646 967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92"
format_stamp: "Formatted at 2026-09-24 19:18:20 on dist-test-slave-0lgr"
server_key: "4e9935bb8a6444342aa1812d9aac6d3e"
server_key_iv: "1fe1c1a669afcd1fa553c8d5307b583d"
server_key_version: "encryptionkey@0"
I20260924 19:18:20.062976 967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 19:18:20.066851 982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:20.068367 967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:18:20.068512 967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92"
format_stamp: "Formatted at 2026-09-24 19:18:20 on dist-test-slave-0lgr"
server_key: "4e9935bb8a6444342aa1812d9aac6d3e"
server_key_iv: "1fe1c1a669afcd1fa553c8d5307b583d"
server_key_version: "encryptionkey@0"
I20260924 19:18:20.068686 967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:20.087486 967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:20.092485 967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:20.092757 967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:20.112875 967 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:45587
I20260924 19:18:20.112967 1033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:45587 every 8 connection(s)
I20260924 19:18:20.114473 967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:20.117463 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 967
I20260924 19:18:20.117631 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:20.118079 11337 external_mini_cluster.cc:1428] Setting key 64b31f91a04e6e1e008bab07b0864714
I20260924 19:18:20.119000 1034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:18:20 dist-test-slave-0lgr krb5kdc[954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:18:20.134487 1034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92: Bootstrap starting.
I20260924 19:18:20.137993 1034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:20.139421 1034 log.cc:824] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:20.143893 1034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92: No bootstrap required, opened a new log
I20260924 19:18:20.143857 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:20.120005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42755 (local address 127.11.18.126:45587)
0924 19:18:20.120819 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:20.120832 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:20.120881 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:18:20.122514 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:20.122536 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:20.122944 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:20.123326 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:20.125145 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:20.126681 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:20.128519 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:20.129249 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:20.129459 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:20.134586 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:20.134595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:20.134611 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:20.134698 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:20.138021 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:20.139002 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:20.139019 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:20.139027 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:20.139216 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:20.139703 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:20.139709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:20.139712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:20.140647 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:20.140916 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:20.141920 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0924 19:18:20.142440 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":390,"threads_started":1}
I20260924 19:18:20.148506 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:20.152729 1034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45587 } }
I20260924 19:18:20.153247 1034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:20.153452 1034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e99e9a4a3c4c998ab8fb8b445c7e92, State: Initialized, Role: FOLLOWER
I20260924 19:18:20.154181 1034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45587 } }
I20260924 19:18:20.154498 1034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:20.154661 1034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:20.154853 1034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:20.157851 1034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45587 } }
I20260924 19:18:20.158411 1034 leader_election.cc:302] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78e99e9a4a3c4c998ab8fb8b445c7e92; no voters:
I20260924 19:18:20.159121 1034 leader_election.cc:288] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:20.159360 1044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:20.160874 1044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [term 1 LEADER]: Becoming Leader. State: Replica: 78e99e9a4a3c4c998ab8fb8b445c7e92, State: Running, Role: LEADER
I20260924 19:18:20.161602 1044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45587 } }
I20260924 19:18:20.161761 1034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:20.165088 1045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45587 } } }
I20260924 19:18:20.165547 1045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:20.166244 1046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e99e9a4a3c4c998ab8fb8b445c7e92. Latest consensus state: current_term: 1 leader_uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e99e9a4a3c4c998ab8fb8b445c7e92" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45587 } } }
I20260924 19:18:20.166695 1046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:20.169987 1050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:20.171886 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:20.153150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44043 (local address 127.11.18.126:45587)
0924 19:18:20.153381 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:20.153387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:20.153869 (+ 482us) server_negotiation.cc:407] Connection header received
0924 19:18:20.154055 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:20.154060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:20.154161 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:20.154282 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:20.155949 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:20.157070 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:20.158826 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:20.159667 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:20.160490 (+ 823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:20.163180 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:20.163189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:20.163197 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:20.163254 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:20.168654 (+ 5400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:20.169720 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:20.169725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:20.169729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:20.169884 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:20.170330 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:20.170341 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:20.170343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:20.170638 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:20.170924 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:20.171268 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 19:18:20.171494 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:18:20.177991 1050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:20.190723 1050 catalog_manager.cc:1347] Generated new cluster ID: f3c6427ecdea4cd497564a272b3089eb
I20260924 19:18:20.190867 1050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:20.202627 1050 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:20.204267 1050 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:20.209518 1050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78e99e9a4a3c4c998ab8fb8b445c7e92: Generated new TSK 0
I20260924 19:18:20.210543 1050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:20.222550 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 967
I20260924 19:18:20.223590 1029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:20.319912 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 967
2026-09-24T19:18:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2113 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:20 dist-test-slave-0lgr krb5kdc[1066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:20 dist-test-slave-0lgr krb5kdc[1066](info): set up 2 sockets
Sep 24 19:18:20 dist-test-slave-0lgr krb5kdc[1066](info): commencing operation
krb5kdc: starting...
W20260924 19:18:21.478098 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:21 dist-test-slave-0lgr krb5kdc[1066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:18:21.758728 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45023
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:22.099400 1079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:22.099762 1079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:22.099936 1079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:22.107528 1079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:22.107699 1079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:22.107807 1079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:22.107934 1079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:22.108013 1079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:22.119024 1079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:22.120455 1079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:22.121862 1079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:22.134852 1087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:22.134882 1085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:22.134922 1084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:22.136035 1079 server_base.cc:1034] running on GCE node
I20260924 19:18:22.136895 1079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:22.138545 1079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:22.139804 1079 hybrid_clock.cc:648] HybridClock initialized: now 1790277502139747 us; error 45 us; skew 500 ppm
I20260924 19:18:22.140200 1079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:22 dist-test-slave-0lgr krb5kdc[1066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277502, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:22.144239 1079 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:18:22.146293 1079 webserver.cc:460] Webserver started at http://127.11.18.126:36299/ using document root <none> and password file <none>
I20260924 19:18:22.147075 1079 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:22.147189 1079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:22.147524 1079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:22.150810 1079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "0b432d6f7d1f48bcb339cedc53737bfc"
format_stamp: "Formatted at 2026-09-24 19:18:22 on dist-test-slave-0lgr"
server_key: "12d2ee32ab3802f1b398b97aac9159d3"
server_key_iv: "bf11aabe8d7e3f9669aa7779b12e035a"
server_key_version: "encryptionkey@0"
I20260924 19:18:22.151670 1079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "0b432d6f7d1f48bcb339cedc53737bfc"
format_stamp: "Formatted at 2026-09-24 19:18:22 on dist-test-slave-0lgr"
server_key: "12d2ee32ab3802f1b398b97aac9159d3"
server_key_iv: "bf11aabe8d7e3f9669aa7779b12e035a"
server_key_version: "encryptionkey@0"
I20260924 19:18:22.158485 1079 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:18:22.163024 1094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:22.164748 1079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:18:22.164959 1079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "0b432d6f7d1f48bcb339cedc53737bfc"
format_stamp: "Formatted at 2026-09-24 19:18:22 on dist-test-slave-0lgr"
server_key: "12d2ee32ab3802f1b398b97aac9159d3"
server_key_iv: "bf11aabe8d7e3f9669aa7779b12e035a"
server_key_version: "encryptionkey@0"
I20260924 19:18:22.165169 1079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:22.197628 1079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:22.202737 1079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:22.203102 1079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:22.224325 1079 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:45023
I20260924 19:18:22.224349 1145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:45023 every 8 connection(s)
I20260924 19:18:22.225919 1079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:22.226197 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 1079
I20260924 19:18:22.226379 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:22.226821 11337 external_mini_cluster.cc:1428] Setting key 38f8c418811228db99b2935086bb73f9
I20260924 19:18:22.231084 1146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:22.241365 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:22.229129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58469 (local address 127.11.18.126:45023)
0924 19:18:22.230128 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:22.230141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:22.230183 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:18:22.232154 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:22.232177 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:22.232585 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:22.232992 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:22.233618 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:22.235182 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:22.236764 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:22.237586 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:22.237791 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:22.238983 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:22.238991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:22.239006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:22.239562 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:22.239651 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:22.239675 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:22.239802 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:22.240028 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:18:22.240313 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":406,"threads_started":1}
I20260924 19:18:22.246040 1146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc: Bootstrap starting.
I20260924 19:18:22.246380 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:22.250753 1146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:22.252442 1146 log.cc:824] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:22.256611 1146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc: No bootstrap required, opened a new log
I20260924 19:18:22.260480 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:22.252526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43855 (local address 127.11.18.126:45023)
0924 19:18:22.252801 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:22.252808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:22.253032 (+ 224us) server_negotiation.cc:407] Connection header received
0924 19:18:22.253403 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:22.253440 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:22.253554 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:22.253690 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:22.254431 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:22.255166 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:22.257404 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:22.257985 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:22.258113 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:22.259582 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:22.259587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:22.259591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:22.259615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:22.259659 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:22.259677 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:22.259760 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:22.259944 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 19:18:22.260030 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 19:18:22.266340 1146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b432d6f7d1f48bcb339cedc53737bfc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45023 } }
I20260924 19:18:22.266815 1146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:22.266983 1146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b432d6f7d1f48bcb339cedc53737bfc, State: Initialized, Role: FOLLOWER
I20260924 19:18:22.267791 1146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b432d6f7d1f48bcb339cedc53737bfc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45023 } }
I20260924 19:18:22.268051 1146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:22.268162 1146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:22.268304 1146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:22.270834 1146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b432d6f7d1f48bcb339cedc53737bfc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45023 } }
I20260924 19:18:22.271381 1146 leader_election.cc:302] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b432d6f7d1f48bcb339cedc53737bfc; no voters:
I20260924 19:18:22.272086 1146 leader_election.cc:288] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:22.272218 1156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:22.273693 1156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [term 1 LEADER]: Becoming Leader. State: Replica: 0b432d6f7d1f48bcb339cedc53737bfc, State: Running, Role: LEADER
I20260924 19:18:22.274264 1156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b432d6f7d1f48bcb339cedc53737bfc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45023 } }
I20260924 19:18:22.274677 1146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:22.277958 1158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b432d6f7d1f48bcb339cedc53737bfc. Latest consensus state: current_term: 1 leader_uuid: "0b432d6f7d1f48bcb339cedc53737bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b432d6f7d1f48bcb339cedc53737bfc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45023 } } }
I20260924 19:18:22.277894 1157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b432d6f7d1f48bcb339cedc53737bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b432d6f7d1f48bcb339cedc53737bfc" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45023 } } }
I20260924 19:18:22.278414 1158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:22.278424 1157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:22.281736 1163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:22.288075 1163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:22.298743 1163 catalog_manager.cc:1347] Generated new cluster ID: 640772f01ba349a7b3b3d2da34e7272b
I20260924 19:18:22.298933 1163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:22.321377 1163 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:22.322882 1163 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:22.334767 1163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b432d6f7d1f48bcb339cedc53737bfc: Generated new TSK 0
I20260924 19:18:22.335759 1163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:22.345963 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1079
W20260924 19:18:22.347210 1141 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58469
W20260924 19:18:22.348049 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1079: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:18:22.348241 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 1079
2026-09-24T19:18:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:22 dist-test-slave-0lgr krb5kdc[1177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:22 dist-test-slave-0lgr krb5kdc[1177](info): set up 2 sockets
Sep 24 19:18:22 dist-test-slave-0lgr krb5kdc[1177](info): commencing operation
krb5kdc: starting...
W20260924 19:18:23.531145 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:23 dist-test-slave-0lgr krb5kdc[1177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:18:23.818532 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42403
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:42403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:24.165141 1190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:24.165474 1190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:24.165645 1190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:24.172966 1190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:24.173084 1190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:24.173167 1190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:24.173285 1190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:24.173395 1190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:24.183472 1190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:42403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:42403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:24.184818 1190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:24.186151 1190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:24.198580 1196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:24.198616 1195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:24.198544 1198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:24.199553 1190 server_base.cc:1034] running on GCE node
I20260924 19:18:24.200384 1190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:24.201895 1190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:24.203145 1190 hybrid_clock.cc:648] HybridClock initialized: now 1790277504203090 us; error 28 us; skew 500 ppm
I20260924 19:18:24.203581 1190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:24 dist-test-slave-0lgr krb5kdc[1177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277504, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:24.207409 1190 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:18:24.209061 1190 webserver.cc:460] Webserver started at http://127.11.18.126:43803/ using document root <none> and password file <none>
I20260924 19:18:24.209862 1190 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:24.210027 1190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:24.210356 1190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:24.213754 1190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "2b0e54bc07a34d11b9c4ebe500edd091"
format_stamp: "Formatted at 2026-09-24 19:18:24 on dist-test-slave-0lgr"
server_key: "ab5946fe04212667ef5ca71193d94df0"
server_key_iv: "69127008ddf961446f8b05c139b39c81"
server_key_version: "encryptionkey@0"
I20260924 19:18:24.214581 1190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "2b0e54bc07a34d11b9c4ebe500edd091"
format_stamp: "Formatted at 2026-09-24 19:18:24 on dist-test-slave-0lgr"
server_key: "ab5946fe04212667ef5ca71193d94df0"
server_key_iv: "69127008ddf961446f8b05c139b39c81"
server_key_version: "encryptionkey@0"
I20260924 19:18:24.220227 1190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:18:24.224210 1205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:24.225930 1190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:18:24.226150 1190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "2b0e54bc07a34d11b9c4ebe500edd091"
format_stamp: "Formatted at 2026-09-24 19:18:24 on dist-test-slave-0lgr"
server_key: "ab5946fe04212667ef5ca71193d94df0"
server_key_iv: "69127008ddf961446f8b05c139b39c81"
server_key_version: "encryptionkey@0"
I20260924 19:18:24.226387 1190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:24.232862 1190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:24.237457 1190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:24.237746 1190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:24.261755 1190 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:42403
I20260924 19:18:24.261798 1256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:42403 every 8 connection(s)
I20260924 19:18:24.263336 1190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:24.265604 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 1190
I20260924 19:18:24.265786 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:24.266212 11337 external_mini_cluster.cc:1428] Setting key 81736cd42e0b0c4dc5768d3bb9f367da
I20260924 19:18:24.267737 1257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:24.274792 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:24.268222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37341 (local address 127.11.18.126:42403)
0924 19:18:24.269156 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:24.269168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:24.269210 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:18:24.270989 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:24.271012 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:24.271386 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:24.271742 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:24.272889 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:24.272896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:24.272909 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:24.272926 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:24.273021 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:24.273045 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:24.273250 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:24.273517 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 19:18:24.273756 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":426,"threads_started":1}
I20260924 19:18:24.278939 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:24.281608 1257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091: Bootstrap starting.
I20260924 19:18:24.285101 1257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:24.286260 1257 log.cc:824] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:24.287436 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:24.284458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50909 (local address 127.11.18.126:42403)
0924 19:18:24.284624 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:24.284629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:24.285136 (+ 507us) server_negotiation.cc:407] Connection header received
0924 19:18:24.285442 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:24.285448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:24.285503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:24.285638 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:24.286704 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:24.286710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:24.286724 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:18:24.286755 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:24.286804 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:18:24.286821 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:24.286968 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:24.287063 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 19:18:24.287136 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:18:24.289659 1257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091: No bootstrap required, opened a new log
I20260924 19:18:24.298122 1257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0e54bc07a34d11b9c4ebe500edd091" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42403 } }
I20260924 19:18:24.298520 1257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:18:24.298659 1257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0e54bc07a34d11b9c4ebe500edd091, State: Initialized, Role: FOLLOWER
I20260924 19:18:24.299319 1257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0e54bc07a34d11b9c4ebe500edd091" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42403 } }
I20260924 19:18:24.299558 1257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:18:24.299692 1257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:18:24.299873 1257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:24.302356 1257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0e54bc07a34d11b9c4ebe500edd091" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42403 } }
I20260924 19:18:24.302942 1257 leader_election.cc:302] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b0e54bc07a34d11b9c4ebe500edd091; no voters:
I20260924 19:18:24.303714 1257 leader_election.cc:288] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:18:24.303870 1267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:24.304725 1267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [term 1 LEADER]: Becoming Leader. State: Replica: 2b0e54bc07a34d11b9c4ebe500edd091, State: Running, Role: LEADER
I20260924 19:18:24.305431 1267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0e54bc07a34d11b9c4ebe500edd091" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42403 } }
I20260924 19:18:24.306097 1257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:24.308312 1268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b0e54bc07a34d11b9c4ebe500edd091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0e54bc07a34d11b9c4ebe500edd091" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42403 } } }
I20260924 19:18:24.308450 1269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b0e54bc07a34d11b9c4ebe500edd091. Latest consensus state: current_term: 1 leader_uuid: "2b0e54bc07a34d11b9c4ebe500edd091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0e54bc07a34d11b9c4ebe500edd091" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 42403 } } }
I20260924 19:18:24.308709 1268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:24.309432 1269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:24.310415 1273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:24.315618 1273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:24.325505 1273 catalog_manager.cc:1347] Generated new cluster ID: 8db074f476d34c0ca61c3834558f0a98
I20260924 19:18:24.325644 1273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:24.359156 1273 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:24.359670 1273 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:24.370898 1273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b0e54bc07a34d11b9c4ebe500edd091: Generated new TSK 0
I20260924 19:18:24.371951 1273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:24.382339 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1190
W20260924 19:18:24.383482 1252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37341
W20260924 19:18:24.384282 11337 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1190: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:18:24.384457 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 1190
2026-09-24T19:18:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2036 ms)
[----------] 18 tests from AuthTokenIssuingTest (36968 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:24 dist-test-slave-0lgr krb5kdc[1288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:24 dist-test-slave-0lgr krb5kdc[1288](info): set up 2 sockets
Sep 24 19:18:24 dist-test-slave-0lgr krb5kdc[1288](info): commencing operation
krb5kdc: starting...
W20260924 19:18:25.542363 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:25 dist-test-slave-0lgr krb5kdc[1288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:18:25.817876 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:46653
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:46653,127.11.18.125:44361,127.11.18.124:36761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:26.164433 1301 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:18:26.164832 1301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:26.164937 1301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:26.165093 1301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:26.172782 1301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:26.172880 1301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:26.172986 1301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:26.173065 1301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:26.173167 1301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:26.183318 1301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:46653,127.11.18.125:44361,127.11.18.124:36761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:26.184676 1301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:26.186038 1301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:26.198218 1306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:26.198267 1309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:26.198187 1307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:26.199131 1301 server_base.cc:1034] running on GCE node
I20260924 19:18:26.199929 1301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:26.201669 1301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:26.202970 1301 hybrid_clock.cc:648] HybridClock initialized: now 1790277506202886 us; error 43 us; skew 500 ppm
I20260924 19:18:26.203419 1301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:26 dist-test-slave-0lgr krb5kdc[1288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277506, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:26.211419 1301 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:18:26.213114 1301 webserver.cc:460] Webserver started at http://127.11.18.126:36031/ using document root <none> and password file <none>
I20260924 19:18:26.213937 1301 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:26.214074 1301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:26.214411 1301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:26.217581 1301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "6d9ccc1f34414380a5368b98c5bbc32e"
format_stamp: "Formatted at 2026-09-24 19:18:26 on dist-test-slave-0lgr"
server_key: "7611096ff754546361c6fb858c23ad3d"
server_key_iv: "ceb154e32b3f8b1fc98b7fcb81b72062"
server_key_version: "encryptionkey@0"
I20260924 19:18:26.218456 1301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "6d9ccc1f34414380a5368b98c5bbc32e"
format_stamp: "Formatted at 2026-09-24 19:18:26 on dist-test-slave-0lgr"
server_key: "7611096ff754546361c6fb858c23ad3d"
server_key_iv: "ceb154e32b3f8b1fc98b7fcb81b72062"
server_key_version: "encryptionkey@0"
I20260924 19:18:26.223950 1301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:18:26.227698 1316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:26.229457 1301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:18:26.229671 1301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "6d9ccc1f34414380a5368b98c5bbc32e"
format_stamp: "Formatted at 2026-09-24 19:18:26 on dist-test-slave-0lgr"
server_key: "7611096ff754546361c6fb858c23ad3d"
server_key_iv: "ceb154e32b3f8b1fc98b7fcb81b72062"
server_key_version: "encryptionkey@0"
I20260924 19:18:26.229936 1301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:26.259588 1301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:26.264179 1301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:26.264483 1301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:26.283823 1301 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:46653
I20260924 19:18:26.283857 1367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:46653 every 8 connection(s)
I20260924 19:18:26.285214 1301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:26.289069 1368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:26.294914 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 1301
I20260924 19:18:26.295131 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:26.295812 11337 external_mini_cluster.cc:1428] Setting key 5c3b2345dd7e7e494becd1afa6098717
I20260924 19:18:26.295866 1368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:26 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277506, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.125@KRBTEST.COM" created.
I20260924 19:18:26.322423 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.299229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.11.18.126:46653)
0924 19:18:26.300098 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:26.300114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:26.300669 (+ 555us) server_negotiation.cc:407] Connection header received
0924 19:18:26.302827 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:26.302838 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:26.303162 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:26.303387 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:26.305141 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.306647 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.308889 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.309714 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.309920 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:26.314250 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:26.314259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:26.314272 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:26.314311 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:26.317579 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:26.318329 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:26.318345 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:26.318348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:26.318491 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:26.318928 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:26.318935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:26.318944 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:26.319465 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:26.319690 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:26.320327 (+ 637us) server_negotiation.cc:299] Negotiation successful
0924 19:18:26.320744 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":611,"thread_start_us":369,"threads_started":1}
I20260924 19:18:26.322456 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.299229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46653 (local address 127.0.0.1:35328)
0924 19:18:26.300494 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.300534 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:26.302377 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:26.303431 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:26.303437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:26.304335 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:26.304868 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:26.304897 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.306826 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.306834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:26.308714 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:26.308732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.309968 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.309978 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:26.310206 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:26.310949 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:26.310985 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:26.313991 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:26.317721 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:26.317735 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:26.317749 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:26.318176 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:26.318622 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:26.318627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:26.318631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:26.318796 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:26.319619 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:26.319634 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:26.319920 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 19:18:26.320192 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 19:18:26.320884 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":159,"spinlock_wait_cycles":61312,"thread_start_us":793,"threads_started":1}
I20260924 19:18:26.325980 1368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:18:26.327330 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.326815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44500)
0924 19:18:26.327033 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.327169 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 19:18:26.327641 1318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
I20260924 19:18:26.330732 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.330230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44516)
0924 19:18:26.330449 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.330548 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 19:18:26.331029 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:44361: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab.
Entry for principal kudu/127.11.18.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab.
I20260924 19:18:26.357081 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.125:44361
--webserver_interface=127.11.18.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab
--principal=kudu/127.11.18.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:46653,127.11.18.125:44361,127.11.18.124:36761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:18:26.380411 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } attempt: 1
I20260924 19:18:26.381772 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.381255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44526)
0924 19:18:26.381514 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.381638 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 19:18:26.382951 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.382535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44542)
0924 19:18:26.382742 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.382818 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 19:18:26.383338 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:44361: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
I20260924 19:18:26.451655 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } attempt: 2
I20260924 19:18:26.452935 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.452424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44554)
0924 19:18:26.452667 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.452759 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:18:26.454046 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.453704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44556)
0924 19:18:26.453868 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.453932 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:18:26.454308 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:44361: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
I20260924 19:18:26.545603 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } attempt: 3
I20260924 19:18:26.546788 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.546351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44564)
0924 19:18:26.546550 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.546650 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 19:18:26.547930 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.547508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44568)
0924 19:18:26.547657 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.547728 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 19:18:26.548120 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:44361: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
I20260924 19:18:26.691394 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } attempt: 4
I20260924 19:18:26.692655 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.692114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44582)
0924 19:18:26.692429 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.692514 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 19:18:26.693691 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.693314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44590)
0924 19:18:26.693485 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.693570 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:18:26.693903 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:44361: Network error: Client connection negotiation failed: client connection to 127.11.18.125:44361: connect: Connection refused (error 111)
W20260924 19:18:26.695844 1375 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:18:26.696182 1375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:26.696280 1375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:26.696410 1375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:26.703855 1375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:26.703944 1375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:26.704031 1375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:26.704128 1375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:26.704233 1375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:26.714821 1375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:46653,127.11.18.125:44361,127.11.18.124:36761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab
--principal=kudu/127.11.18.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.125:44361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.18.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:26.716177 1375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:26.717542 1375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:26.730624 1380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:26.730628 1381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:26.731140 1383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:26.732478 1375 server_base.cc:1034] running on GCE node
I20260924 19:18:26.733448 1375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:26.735049 1375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:26.736409 1375 hybrid_clock.cc:648] HybridClock initialized: now 1790277506736343 us; error 44 us; skew 500 ppm
I20260924 19:18:26.736732 1375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:26 dist-test-slave-0lgr krb5kdc[1288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277506, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:26.740259 1375 init.cc:377] Logged in from keytab as kudu/127.11.18.125@KRBTEST.COM (short username kudu)
I20260924 19:18:26.742131 1375 webserver.cc:460] Webserver started at http://127.11.18.125:42569/ using document root <none> and password file <none>
I20260924 19:18:26.742911 1375 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:26.743036 1375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:26.743441 1375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:26.746812 1375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/data/instance:
uuid: "0d27713e25f14513b68a653b4d75e0ae"
format_stamp: "Formatted at 2026-09-24 19:18:26 on dist-test-slave-0lgr"
server_key: "37550b755783cc5a3dcb1c497c7cf853"
server_key_iv: "07c16a7d42f811e502d1ce7db1c4a2ed"
server_key_version: "encryptionkey@0"
I20260924 19:18:26.747689 1375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/wal/instance:
uuid: "0d27713e25f14513b68a653b4d75e0ae"
format_stamp: "Formatted at 2026-09-24 19:18:26 on dist-test-slave-0lgr"
server_key: "37550b755783cc5a3dcb1c497c7cf853"
server_key_iv: "07c16a7d42f811e502d1ce7db1c4a2ed"
server_key_version: "encryptionkey@0"
I20260924 19:18:26.753568 1375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 19:18:26.757700 1390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:26.759222 1375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:18:26.759466 1375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/wal
uuid: "0d27713e25f14513b68a653b4d75e0ae"
format_stamp: "Formatted at 2026-09-24 19:18:26 on dist-test-slave-0lgr"
server_key: "37550b755783cc5a3dcb1c497c7cf853"
server_key_iv: "07c16a7d42f811e502d1ce7db1c4a2ed"
server_key_version: "encryptionkey@0"
I20260924 19:18:26.759711 1375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:26.770846 1375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:26.775178 1375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:26.775492 1375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:26.795014 1375 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.125:44361
I20260924 19:18:26.795039 1441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.125:44361 every 8 connection(s)
I20260924 19:18:26.796561 1375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
I20260924 19:18:26.800534 1442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:26.803460 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 1375
I20260924 19:18:26.803646 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-1/wal/instance
I20260924 19:18:26.804258 11337 external_mini_cluster.cc:1428] Setting key 1d7f215f7da9e67017e136635656d279
I20260924 19:18:26.811020 1442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:26 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277506, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.124@KRBTEST.COM" created.
I20260924 19:18:26.837715 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.815114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.11.18.126:46653)
0924 19:18:26.815365 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:26.815372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:26.816815 (+ 1443us) server_negotiation.cc:407] Connection header received
0924 19:18:26.818961 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:26.818967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:26.819073 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:26.819205 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:26.820846 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.821730 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.823516 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.824062 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.824234 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:26.828705 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:26.828717 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:26.828725 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:26.828782 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:26.833389 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:26.834498 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:26.834503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:26.834506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:26.834645 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:26.835191 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:26.835200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:26.835207 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:26.835593 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:26.835747 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:26.836865 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0924 19:18:26.837178 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 19:18:26.838333 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.815672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46653 (local address 127.0.0.1:35344)
0924 19:18:26.816565 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.816619 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:26.818648 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:26.819453 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:26.819462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:26.820030 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:26.820642 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:26.820670 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.821908 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.821912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:26.823316 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:26.823330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.824341 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.824349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:26.824610 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:26.825460 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:26.825486 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:26.828387 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:26.833570 (+ 5183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:26.833578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:26.833594 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:26.834129 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:26.834780 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:26.834785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:26.834787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:26.834991 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:26.835753 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:26.835760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:26.836304 (+ 544us) client_negotiation.cc:769] Sending connection context
0924 19:18:26.836673 (+ 369us) client_negotiation.cc:240] Negotiation successful
0924 19:18:26.837104 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":399,"threads_started":1}
I20260924 19:18:26.840236 1442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:26 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277506, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:18:26.858721 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.841210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44592)
0924 19:18:26.841411 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.841432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:26.841552 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:26.842715 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:26.842718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:26.843117 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:26.843255 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:26.843263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.845645 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.845649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:26.846701 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:26.846714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.848075 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.848088 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:26.848320 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:26.849055 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:26.849087 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:26.852003 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:26.855842 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:26.855850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:26.855855 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:26.856256 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:26.856680 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:26.856685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:26.856689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:26.856846 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:26.857630 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:26.857635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:26.857806 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 19:18:26.857959 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 19:18:26.858272 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 19:18:26.858807 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.841201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.11.18.125:44361)
0924 19:18:26.842128 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:26.842137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:26.842195 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:18:26.842267 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:26.842278 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:26.842542 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:26.842720 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:26.843443 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.845468 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.846865 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.847880 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.847995 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:26.852269 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:26.852281 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:26.852296 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:26.852331 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:26.855681 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:26.856402 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:26.856418 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:26.856421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:26.856544 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:26.856968 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:26.856973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:26.856984 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:26.857481 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:26.857721 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:26.858110 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 19:18:26.858389 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":689,"thread_start_us":536,"threads_started":1}
I20260924 19:18:26.861372 1442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:18:26.862797 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.862100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53456)
0924 19:18:26.862425 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.862564 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
W20260924 19:18:26.863178 1392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
I20260924 19:18:26.865224 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.864790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53470)
0924 19:18:26.864983 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.865085 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 19:18:26.865612 1442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab.
Entry for principal kudu/127.11.18.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab.
I20260924 19:18:26.874547 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.124:36761
--webserver_interface=127.11.18.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab
--principal=kudu/127.11.18.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:46653,127.11.18.125:44361,127.11.18.124:36761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:18:26.915016 1442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 1
I20260924 19:18:26.916229 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.915756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53472)
0924 19:18:26.915957 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.916065 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 19:18:26.917481 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.917009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53478)
0924 19:18:26.917206 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.917298 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 19:18:26.917733 1442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
I20260924 19:18:26.986004 1442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 2
I20260924 19:18:26.987291 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.986812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53480)
0924 19:18:26.987026 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.987127 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:18:26.988346 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.987993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53490)
0924 19:18:26.988149 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.988231 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:18:26.988572 1442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
I20260924 19:18:26.993366 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } attempt: 5
Sep 24 19:18:26 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277506, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:18:27.008561 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.994177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.11.18.125:44361)
0924 19:18:26.994463 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:26.994470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:26.994495 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:18:26.994576 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:26.994579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:26.994642 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:26.994738 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:26.995682 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.996429 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.998127 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.998640 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.998782 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.002525 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.002531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.002534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.002569 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.005722 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.006491 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.006499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.006502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.006635 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.007047 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.007054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.007058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.007366 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.007536 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.007949 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.008182 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260924 19:18:27.008567 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:26.994061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44594)
0924 19:18:26.994280 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:26.994303 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:26.994462 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:26.994797 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:26.994802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:26.995283 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:26.995503 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:26.995515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.996573 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.996579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:26.997948 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:26.997963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:26.998799 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:26.998805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:26.999016 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:26.999725 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:26.999748 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:27.002306 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:27.005881 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.005887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.005892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.006357 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.006761 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.006768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.006772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.006927 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.007495 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:27.007501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:27.007679 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 19:18:27.007838 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 19:18:27.008159 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 19:18:27.009759 1368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:18:27.010815 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.010398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53500)
0924 19:18:27.010571 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.010667 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:18:27.011780 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.011423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53512)
0924 19:18:27.011575 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.011677 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 19:18:27.012033 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
I20260924 19:18:27.063251 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 1
I20260924 19:18:27.064703 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.064142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53524)
0924 19:18:27.064440 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.064530 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260924 19:18:27.065840 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.065460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53532)
0924 19:18:27.065598 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.065679 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 19:18:27.066114 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
I20260924 19:18:27.079803 1442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 3
I20260924 19:18:27.081089 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.080570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53548)
0924 19:18:27.080819 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.080908 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:18:27.082165 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.081752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53556)
0924 19:18:27.081977 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.082052 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260924 19:18:27.082376 1442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
I20260924 19:18:27.134413 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 2
I20260924 19:18:27.135859 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.135262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53568)
0924 19:18:27.135570 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.135682 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 19:18:27.136909 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.136537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53580)
0924 19:18:27.136697 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.136786 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:18:27.137151 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
W20260924 19:18:27.216481 1449 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:18:27.216882 1449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:27.217010 1449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:27.217167 1449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 19:18:27.225605 1442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 4
I20260924 19:18:27.227149 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.226494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53596)
0924 19:18:27.226787 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.226922 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 19:18:27.227339 1449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:27.227455 1449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:27.227548 1449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:27.227612 1449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:27.227689 1449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:27.228190 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.227834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53608)
0924 19:18:27.227988 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.228084 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:18:27.228451 1442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
I20260924 19:18:27.247439 1449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:46653,127.11.18.125:44361,127.11.18.124:36761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab
--principal=kudu/127.11.18.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.124:36761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.18.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:27.249382 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 3
I20260924 19:18:27.249698 1449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:27.250703 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.250097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53612)
0924 19:18:27.250364 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.250483 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 19:18:27.251708 1449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:18:27.251812 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.251433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53618)
0924 19:18:27.251577 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.251670 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 19:18:27.252012 1368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:36761: Network error: Client connection negotiation failed: client connection to 127.11.18.124:36761: connect: Connection refused (error 111)
W20260924 19:18:27.270041 1455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:27.270071 1457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:27.270176 1454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:27.270973 1449 server_base.cc:1034] running on GCE node
I20260924 19:18:27.271977 1449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:27.274008 1449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:27.275297 1449 hybrid_clock.cc:648] HybridClock initialized: now 1790277507275208 us; error 68 us; skew 500 ppm
I20260924 19:18:27.275779 1449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277507, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:27.280222 1449 init.cc:377] Logged in from keytab as kudu/127.11.18.124@KRBTEST.COM (short username kudu)
I20260924 19:18:27.281958 1449 webserver.cc:460] Webserver started at http://127.11.18.124:38597/ using document root <none> and password file <none>
I20260924 19:18:27.282677 1449 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:27.282814 1449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:27.283226 1449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:27.287242 1449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/data/instance:
uuid: "ff70589bba3647ada19aa2cb0de6569c"
format_stamp: "Formatted at 2026-09-24 19:18:27 on dist-test-slave-0lgr"
server_key: "e65881eeb9cba1ec1e963cb33381d2cb"
server_key_iv: "913e207564aebde77f54139e93781ff1"
server_key_version: "encryptionkey@0"
I20260924 19:18:27.288295 1449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/wal/instance:
uuid: "ff70589bba3647ada19aa2cb0de6569c"
format_stamp: "Formatted at 2026-09-24 19:18:27 on dist-test-slave-0lgr"
server_key: "e65881eeb9cba1ec1e963cb33381d2cb"
server_key_iv: "913e207564aebde77f54139e93781ff1"
server_key_version: "encryptionkey@0"
I20260924 19:18:27.295110 1449 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 19:18:27.298889 1464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:27.300356 1449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:18:27.300554 1449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/wal
uuid: "ff70589bba3647ada19aa2cb0de6569c"
format_stamp: "Formatted at 2026-09-24 19:18:27 on dist-test-slave-0lgr"
server_key: "e65881eeb9cba1ec1e963cb33381d2cb"
server_key_iv: "913e207564aebde77f54139e93781ff1"
server_key_version: "encryptionkey@0"
I20260924 19:18:27.300793 1449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:27.317549 1449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:27.322328 1449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:27.322636 1449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:27.342188 1449 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.124:36761
I20260924 19:18:27.342208 1515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.124:36761 every 8 connection(s)
I20260924 19:18:27.343617 1449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
I20260924 19:18:27.347519 1516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:27.352371 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 1449
I20260924 19:18:27.352568 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277128821086-11337-0/minicluster-data/master-2/wal/instance
I20260924 19:18:27.353150 11337 external_mini_cluster.cc:1428] Setting key cc72abc493e18bc634bc169919abf8e1
I20260924 19:18:27.355775 1516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277507, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
I20260924 19:18:27.376253 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.358526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.11.18.126:46653)
0924 19:18:27.359776 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.359782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.359802 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:18:27.359865 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.359872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.359958 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.360088 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.361725 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.362617 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.364279 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.364766 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.364907 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.368519 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.368524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.368528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.368564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.372272 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.373158 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.373167 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.373173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.373358 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.374025 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.374029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.374031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.374409 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.374581 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.375542 (+ 961us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.375778 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":977,"threads_started":1}
I20260924 19:18:27.377493 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.355198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.11.18.124:36761)
0924 19:18:27.356242 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.356255 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.356298 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:18:27.358242 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.358264 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.358759 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.359302 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.361029 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.362519 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.364096 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.364660 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.364791 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.369528 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.369541 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.369559 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.369595 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.373415 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.374275 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.374284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.374287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.374448 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.374925 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.374929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.374931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.375283 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.375675 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.376046 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.376301 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":768,"spinlock_wait_cycles":15744,"thread_start_us":558,"threads_started":1}
I20260924 19:18:27.377543 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.358437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:46653 (local address 127.0.0.1:35356)
0924 19:18:27.359095 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.359135 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:27.359278 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:27.360174 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:27.360179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:27.360798 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:27.361494 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.361516 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.362790 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.362793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.364143 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.364154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.365009 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.365024 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.365149 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.365975 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:27.365998 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:27.368298 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:27.372436 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.372446 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.372456 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.373023 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.373517 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.373525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.373529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.373857 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.374590 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:27.374599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:27.375111 (+ 512us) client_negotiation.cc:769] Sending connection context
0924 19:18:27.375449 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 19:18:27.376050 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":66,"thread_start_us":302,"threads_started":1}
I20260924 19:18:27.379676 1516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277507, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:18:27.396788 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.380830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:44361 (local address 127.0.0.1:44598)
0924 19:18:27.381208 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.381256 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:27.381569 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:27.382161 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:27.382168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:27.382651 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:27.382864 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.382879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.384269 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.384273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.385599 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.385615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.386529 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.386533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.386708 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.387397 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:27.387423 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:27.389998 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:27.393732 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.393739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.393743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.394312 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.394813 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.394820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.394823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.395018 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.395598 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:27.395604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:27.395783 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 19:18:27.395953 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 19:18:27.396293 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260924 19:18:27.396828 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.380734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.11.18.125:44361)
0924 19:18:27.380950 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.380959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.381398 (+ 439us) server_negotiation.cc:407] Connection header received
0924 19:18:27.381732 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.381740 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.381852 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.381937 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.383062 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.384107 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.385780 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.386373 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.386503 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.390319 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.390324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.390327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.390360 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.393585 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.394478 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.394485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.394489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.394657 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.395148 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.395152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.395154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.395462 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.395633 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.396078 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.396304 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:18:27.398051 1516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:18:27.398686 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.381287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.11.18.126:46653)
0924 19:18:27.381579 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.381588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.381610 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:18:27.381671 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.381678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.381779 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.381885 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.383582 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.384585 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.386286 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.386876 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.387037 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.391708 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.391714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.391717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.391754 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.395327 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.396047 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.396053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.396056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.396199 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.396620 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.396631 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.396633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.397401 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.397749 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.398097 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.398359 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277507, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:18:27.413379 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.398900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53648)
0924 19:18:27.399054 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.399074 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:27.399186 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:27.399790 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:27.399793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:27.400192 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:27.400326 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.400334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.401905 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.401909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.402996 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.403009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.403705 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.403710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.403928 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.404591 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:27.404607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:27.407102 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:27.410502 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.410508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.410512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.411092 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.411529 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.411532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.411535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.411664 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.412413 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:27.412417 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:27.412551 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:18:27.412666 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 19:18:27.412999 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 19:18:27.413930 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.399405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.11.18.124:36761)
0924 19:18:27.399563 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.399567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.399580 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:18:27.399615 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.399619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.399667 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.399773 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.400485 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.401730 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.403159 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.403550 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.403698 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.407306 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.407313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.407318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.407359 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.410348 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.411227 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.411232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.411235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.411387 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.411828 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.411838 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.411841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.412225 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.412389 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.412819 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.413224 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":59}
I20260924 19:18:27.417287 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.400495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.11.18.125:44361)
0924 19:18:27.400656 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.400660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.400776 (+ 116us) server_negotiation.cc:407] Connection header received
0924 19:18:27.400929 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.400932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.400985 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.401057 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.402642 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.403403 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.405083 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.405812 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.405977 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.411043 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.411057 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.411060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.411095 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.414591 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.415293 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.415299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.415303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.415455 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.415854 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.415859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.415863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.416131 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.416435 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.416690 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.416934 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:18:27.419193 1516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: Bootstrap starting.
I20260924 19:18:27.422714 1516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:27.422798 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:27.423756 1516 log.cc:824] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:27.426944 1516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: No bootstrap required, opened a new log
I20260924 19:18:27.426179 1343 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6d9ccc1f34414380a5368b98c5bbc32e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35362
I20260924 19:18:27.429466 1368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 4
I20260924 19:18:27.437480 1516 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } }
I20260924 19:18:27.437960 1516 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff70589bba3647ada19aa2cb0de6569c, State: Initialized, Role: FOLLOWER
I20260924 19:18:27.438568 1516 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } }
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277506, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
I20260924 19:18:27.439801 1523 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.440213 1523 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:27.441098 1516 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:27.447348 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.430205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.11.18.124:36761)
0924 19:18:27.430376 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.430381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.430492 (+ 111us) server_negotiation.cc:407] Connection header received
0924 19:18:27.430688 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.430693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.430778 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.430878 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.431962 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.433154 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.435270 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.436029 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.436174 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.440510 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.440516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.440521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.440565 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.444259 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.445106 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.445115 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.445118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.445275 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.445834 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.445843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.445846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.446192 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.446370 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.446750 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.446975 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:18:27.447361 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.430125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53652)
0924 19:18:27.430360 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.430386 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:27.430532 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:27.430938 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:27.430943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:27.431529 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:27.431789 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.431805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.433407 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.433414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.435099 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.435121 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.436201 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.436209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.436431 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.437189 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:27.437217 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:27.440241 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:27.444400 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.444408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.444414 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.444988 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.445533 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.445540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.445545 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.445720 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.446332 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:27.446336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:27.446503 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 19:18:27.446650 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 19:18:27.446952 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 19:18:27.452857 1368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e: Bootstrap starting.
I20260924 19:18:27.457266 1368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:27.458719 1368 log.cc:824] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e: Log is configured to *not* fsync() on all Append() calls
W20260924 19:18:27.462188 1534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:18:27.462613 1534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:18:27.468633 1368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e: No bootstrap required, opened a new log
I20260924 19:18:27.457670 1449 master_runner.cc:186] Error getting master registration for 127.11.18.126:46653: OK, instance_id { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" instance_seqno: 1790277506278178 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:18:27.471722 1449 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:18:27.476778 1368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } }
I20260924 19:18:27.477494 1368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d9ccc1f34414380a5368b98c5bbc32e, State: Initialized, Role: FOLLOWER
I20260924 19:18:27.478163 1368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } }
I20260924 19:18:27.479400 1537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.479843 1537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:27.480629 1368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:27.480688 1343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:18:27.480913 1343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:27.484077 1343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } }
I20260924 19:18:27.485543 1343 leader_election.cc:288] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [CANDIDATE]: Term 1 election: Requested vote from peers 0d27713e25f14513b68a653b4d75e0ae (127.11.18.125:44361), ff70589bba3647ada19aa2cb0de6569c (127.11.18.124:36761)
I20260924 19:18:27.486996 1491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ff70589bba3647ada19aa2cb0de6569c"
I20260924 19:18:27.487535 1491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:27.491571 1491 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d9ccc1f34414380a5368b98c5bbc32e in term 1.
I20260924 19:18:27.492292 1318 leader_election.cc:302] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d9ccc1f34414380a5368b98c5bbc32e, ff70589bba3647ada19aa2cb0de6569c; no voters:
I20260924 19:18:27.492797 1537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:27.493479 1537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [term 1 LEADER]: Becoming Leader. State: Replica: 6d9ccc1f34414380a5368b98c5bbc32e, State: Running, Role: LEADER
I20260924 19:18:27.494127 1537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } }
I20260924 19:18:27.497768 1544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9ccc1f34414380a5368b98c5bbc32e. Latest consensus state: current_term: 1 leader_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.498361 1544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:27.499830 1547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:27.507543 1547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 19:18:27.517604 1322 tablet.cc:2369] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:18:27.527649 1442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } attempt: 5
I20260924 19:18:27.533986 1559 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 24 19:18:27 dist-test-slave-0lgr krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277506, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
I20260924 19:18:27.542372 1491 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [term 1 FOLLOWER]: Refusing update from remote peer 6d9ccc1f34414380a5368b98c5bbc32e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:18:27.543645 1537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:18:27.554544 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.528307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:36761 (local address 127.0.0.1:53660)
0924 19:18:27.528582 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:27.528604 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:27.528755 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:27.529151 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:27.529155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:27.529727 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:27.529950 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.529964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.531497 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.531503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.532934 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:27.532953 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.534481 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.534488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:27.534709 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.535451 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:27.535485 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:27.540206 (+ 4721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:27.548798 (+ 8592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.548805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.548809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.549888 (+ 1079us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.550887 (+ 999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:27.550913 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:27.550916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:27.551271 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:27.552478 (+ 1207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:27.552487 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:27.552733 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 19:18:27.552965 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 19:18:27.553886 (+ 921us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 19:18:27.560403 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.528405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.11.18.124:36761)
0924 19:18:27.528583 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.528591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.528681 (+ 90us) server_negotiation.cc:407] Connection header received
0924 19:18:27.528893 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.528899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.529012 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.529129 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.530101 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.531360 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.533176 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.534008 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.534143 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.540531 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.540539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.540544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.540592 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.548464 (+ 7872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.550084 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.550092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.550096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.550715 (+ 619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.551469 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.551474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.551504 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.552204 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.553258 (+ 1054us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.555651 (+ 2393us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.557678 (+ 2027us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:18:27.566642 1442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae: Bootstrap starting.
I20260924 19:18:27.571017 1442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae: Neither blocks nor log segments found. Creating new log.
W20260924 19:18:27.574883 1470 tablet.cc:2369] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:18:27.576645 1442 log.cc:824] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:27.553499 1301 master_runner.cc:186] Error getting master registration for 127.11.18.125:44361: OK, instance_id { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" instance_seqno: 1790277506789519 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:18:27.580901 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.521705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.11.18.125:44361)
0924 19:18:27.521911 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.521918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.529027 (+ 7109us) server_negotiation.cc:407] Connection header received
0924 19:18:27.544602 (+ 15575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.544608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.544693 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.544766 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.553812 (+ 9046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.555351 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.557939 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.558980 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.559090 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.572752 (+ 13662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.572760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.572765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.572810 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.576948 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.578029 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.578035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.578039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.578217 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.578678 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.578683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.578687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.579338 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.579593 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.580171 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.580484 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:18:27.586495 1301 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:18:27.589219 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.524151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.11.18.126:46653)
0924 19:18:27.524338 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.524344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.530948 (+ 6604us) server_negotiation.cc:407] Connection header received
0924 19:18:27.551436 (+ 20488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.551442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.551524 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.551598 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.562159 (+ 10561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.563241 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.567663 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.568319 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.568450 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.581719 (+ 13269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.581726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.581731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.581783 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.585842 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.586798 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.586806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.586810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.586981 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.587474 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.587479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.587482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.587845 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.588096 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.588640 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.588859 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:18:27.595659 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.531467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.11.18.124:36761)
0924 19:18:27.548121 (+ 16654us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.548127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.548151 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:18:27.548209 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.548214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.548299 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.548416 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.553428 (+ 5012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.560144 (+ 6716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.565790 (+ 5646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.567354 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.568777 (+ 1423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.574096 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:27.574109 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:27.574117 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:27.574266 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:27.583587 (+ 9321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.585386 (+ 1799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.585426 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.585430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.587434 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:27.588378 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:27.588385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:27.588389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:27.589225 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:27.593017 (+ 3792us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.593386 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.594243 (+ 857us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16476,"thread_start_us":17164,"threads_started":1}
I20260924 19:18:27.606700 1442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae: No bootstrap required, opened a new log
I20260924 19:18:27.612927 1547 catalog_manager.cc:1347] Generated new cluster ID: 23413cb4a374433187844672fc938584
I20260924 19:18:27.613050 1547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:27.613147 1523 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9ccc1f34414380a5368b98c5bbc32e. Latest consensus state: current_term: 1 leader_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.613713 1523 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:27.614390 1544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.617357 1544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:27.629495 1442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } }
I20260924 19:18:27.630084 1442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d27713e25f14513b68a653b4d75e0ae, State: Initialized, Role: FOLLOWER
I20260924 19:18:27.630748 1442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } }
I20260924 19:18:27.637710 1523 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.637995 1523 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:27.639873 1442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:27.639937 1417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0d27713e25f14513b68a653b4d75e0ae"
I20260924 19:18:27.640455 1417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:27.637809 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.641817 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:27.646840 1417 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d9ccc1f34414380a5368b98c5bbc32e in term 1.
I20260924 19:18:27.647694 1416 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [term 1 FOLLOWER]: Refusing update from remote peer 6d9ccc1f34414380a5368b98c5bbc32e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:18:27.649736 1537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:18:27.706584 1547 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:27.707464 1578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.709043 1537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.709543 1537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:27.708091 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9ccc1f34414380a5368b98c5bbc32e. Latest consensus state: current_term: 1 leader_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9ccc1f34414380a5368b98c5bbc32e" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 46653 } } peers { permanent_uuid: "0d27713e25f14513b68a653b4d75e0ae" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 44361 } } peers { permanent_uuid: "ff70589bba3647ada19aa2cb0de6569c" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 36761 } } }
I20260924 19:18:27.710124 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:27.710354 1578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:27.738127 1547 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:27.758237 1547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d9ccc1f34414380a5368b98c5bbc32e: Generated new TSK 0
I20260924 19:18:27.759450 1547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:27.768381 1585 catalog_manager.cc:1259] Loaded cluster ID: 23413cb4a374433187844672fc938584
I20260924 19:18:27.768500 1585 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae: loading cluster ID for follower catalog manager: success
I20260924 19:18:27.774142 1585 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae: acquiring CA information for follower catalog manager: success
I20260924 19:18:27.776713 1585 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d27713e25f14513b68a653b4d75e0ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:18:27.892554 1521 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:27.883038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.11.18.126:46653)
0924 19:18:27.883266 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.883273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.890510 (+ 7237us) server_negotiation.cc:407] Connection header received
0924 19:18:27.890955 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.890961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.891048 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.891120 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.892308 (+ 1188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35370: BlockingRecv error: recv got EOF from 127.0.0.1:35370 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 19:18:27.896235 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:27.887723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.11.18.124:36761)
0924 19:18:27.887899 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.887905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.894020 (+ 6115us) server_negotiation.cc:407] Connection header received
0924 19:18:27.894737 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.894744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.894827 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.894900 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.895992 (+ 1092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53680: BlockingRecv error: recv got EOF from 127.0.0.1:53680 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 19:18:27.898269 1446 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:27.874936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.11.18.125:44361)
0924 19:18:27.875163 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.875169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.887009 (+ 11840us) server_negotiation.cc:407] Connection header received
0924 19:18:27.887206 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.887214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.887318 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.887442 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:27.898017 (+ 10575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44630: BlockingRecv error: recv got EOF from 127.0.0.1:44630 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:18:27.956533 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:27.936453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.11.18.125:44361)
0924 19:18:27.936673 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.936680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:27.947096 (+ 10416us) server_negotiation.cc:407] Connection header received
0924 19:18:27.947790 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:27.947799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:27.948065 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:27.948240 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:27.949109 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.950448 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.952859 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:27.953688 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:27.953830 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:27.954733 (+ 903us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:27.955557 (+ 824us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:27.955699 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:27.955992 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 19:18:27.956116 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 19:18:28.005357 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:27.991635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.11.18.124:36761)
0924 19:18:27.991854 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:27.991864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.002722 (+ 10858us) server_negotiation.cc:407] Connection header received
0924 19:18:28.003352 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.003360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.003501 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.003592 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:28.004992 (+ 1400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53682: BlockingRecv error: recv got EOF from 127.0.0.1:53682 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:18:28.062467 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.042863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.11.18.125:44361)
0924 19:18:28.043121 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.043129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.054353 (+ 11224us) server_negotiation.cc:407] Connection header received
0924 19:18:28.055004 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.055011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.055155 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.055258 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.056036 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.057376 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.059795 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.060571 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.060700 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.061474 (+ 774us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.061664 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.061737 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.062001 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.062106 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260924 19:18:28.129643 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:28.116056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.11.18.124:36761)
0924 19:18:28.116255 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.116267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.127486 (+ 11219us) server_negotiation.cc:407] Connection header received
0924 19:18:28.128079 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.128086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.128188 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.128280 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:28.129393 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53694: BlockingRecv error: recv got EOF from 127.0.0.1:53694 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:18:28.185292 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.166992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.11.18.125:44361)
0924 19:18:28.167190 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.167196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.177915 (+ 10719us) server_negotiation.cc:407] Connection header received
0924 19:18:28.178506 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.178513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.178659 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.178737 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.179481 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.180679 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.182842 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.183540 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.183656 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.184304 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.184454 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.184522 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.184843 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.184940 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 19:18:28.238570 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:28.225123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.11.18.124:36761)
0924 19:18:28.225348 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.225354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.236154 (+ 10800us) server_negotiation.cc:407] Connection header received
0924 19:18:28.236688 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.236695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.236878 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.236978 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:28.238241 (+ 1263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53708: BlockingRecv error: recv got EOF from 127.0.0.1:53708 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:18:28.268492 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.260330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.11.18.125:44361)
0924 19:18:28.260532 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.260541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.262171 (+ 1630us) server_negotiation.cc:407] Connection header received
0924 19:18:28.262396 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.262402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.262519 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.262649 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.263255 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.264451 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.266319 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.267014 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.267154 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.267625 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.267785 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.267889 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.268069 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.268162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 19:18:28.278851 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:28.276406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.11.18.124:36761)
0924 19:18:28.276687 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.276694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.277118 (+ 424us) server_negotiation.cc:407] Connection header received
0924 19:18:28.277306 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.277312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.277488 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.277582 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:28.278623 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53716: BlockingRecv error: recv got EOF from 127.0.0.1:53716 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260924 19:18:28.296800 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.290916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.11.18.125:44361)
0924 19:18:28.291132 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.291139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.291539 (+ 400us) server_negotiation.cc:407] Connection header received
0924 19:18:28.291789 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.291795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.291869 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.291981 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.292449 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.293265 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.294777 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.295297 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.295465 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.295903 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.296051 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.296164 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.296331 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.296422 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 19:18:28.311611 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:28.303541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.11.18.124:36761)
0924 19:18:28.303828 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.303838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.309992 (+ 6154us) server_negotiation.cc:407] Connection header received
0924 19:18:28.310241 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.310246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.310338 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.310459 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:28.311326 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53722: BlockingRecv error: recv got EOF from 127.0.0.1:53722 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:18:28.337576 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.331466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.11.18.125:44361)
0924 19:18:28.331658 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.331665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.332023 (+ 358us) server_negotiation.cc:407] Connection header received
0924 19:18:28.332188 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.332194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.332288 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.332401 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.332862 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.334071 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.335497 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.336182 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.336311 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.336701 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.336917 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.337009 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.337163 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.337250 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 19:18:28.358026 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:28.355640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.11.18.124:36761)
0924 19:18:28.355827 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.355833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.356433 (+ 600us) server_negotiation.cc:407] Connection header received
0924 19:18:28.356624 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.356630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.356731 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.356841 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:28.357793 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53732: BlockingRecv error: recv got EOF from 127.0.0.1:53732 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:18:28.398723 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.393371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.11.18.125:44361)
0924 19:18:28.393605 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.393613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.393932 (+ 319us) server_negotiation.cc:407] Connection header received
0924 19:18:28.394161 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.394167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.394273 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.394383 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.394802 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.395506 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.396810 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.397313 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.397513 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.397936 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.398043 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.398144 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.398289 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.398373 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 19:18:28.407539 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:28.405572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.11.18.124:36761)
0924 19:18:28.405816 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.405821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.406306 (+ 485us) server_negotiation.cc:407] Connection header received
0924 19:18:28.406444 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.406448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.406524 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.406617 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:28.407354 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53736: BlockingRecv error: recv got EOF from 127.0.0.1:53736 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:18:28.465512 1534 catalog_manager.cc:1259] Loaded cluster ID: 23413cb4a374433187844672fc938584
I20260924 19:18:28.465618 1534 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: loading cluster ID for follower catalog manager: success
I20260924 19:18:28.468426 1534 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: acquiring CA information for follower catalog manager: success
I20260924 19:18:28.470427 1534 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff70589bba3647ada19aa2cb0de6569c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:18:28.480302 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.474999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.11.18.125:44361)
0924 19:18:28.475190 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.475195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.475592 (+ 397us) server_negotiation.cc:407] Connection header received
0924 19:18:28.475747 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.475751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.475819 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.475916 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.476304 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.477059 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.478598 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.479091 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.479191 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.479602 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.479703 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.479771 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.479998 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.480073 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:18:28.492579 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.486521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.11.18.124:36761)
0924 19:18:28.486839 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.486847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.487116 (+ 269us) server_negotiation.cc:407] Connection header received
0924 19:18:28.487338 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.487356 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.487534 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.487671 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.488117 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.488903 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.490327 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.490741 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.490883 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.491422 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.491966 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.492140 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.492193 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.492279 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 19:18:28.507773 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.499636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.11.18.125:44361)
0924 19:18:28.499953 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.499962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.500874 (+ 912us) server_negotiation.cc:407] Connection header received
0924 19:18:28.501700 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.501706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.501800 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.501903 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.502464 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.503500 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.505714 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.506346 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.506498 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.506968 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.507123 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.507193 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.507413 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.507496 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:18:28.508248 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.501053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.11.18.124:36761)
0924 19:18:28.501225 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.501233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.501701 (+ 468us) server_negotiation.cc:407] Connection header received
0924 19:18:28.501941 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.501947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.502039 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.502145 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.502847 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.503876 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.505989 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.506682 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.506818 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.507330 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.507483 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.507565 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.507901 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.507987 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:18:28.508757 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:28.499905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.11.18.126:46653)
0924 19:18:28.500495 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:28.500503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:28.501556 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 19:18:28.502129 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:28.502134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:28.502368 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:28.502564 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:28.503191 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.504240 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.506186 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:28.506858 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:28.507015 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:28.507606 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:28.508100 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:28.508198 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:28.508449 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 19:18:28.508537 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":286,"threads_started":1}
I20260924 19:18:28.515287 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1301
I20260924 19:18:28.516300 1363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:28.628964 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 1301
W20260924 19:18:28.651535 1392 connection.cc:535] server connection from 127.0.0.1:44594 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:18:28.652024 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1375
I20260924 19:18:28.653122 1437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:28.763899 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 1375
I20260924 19:18:28.784469 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1449
I20260924 19:18:28.785770 1511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:28.894241 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 1449
2026-09-24T19:18:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4513 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:18:29 dist-test-slave-0lgr krb5kdc[1689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:18:29 dist-test-slave-0lgr krb5kdc[1689](info): set up 2 sockets
Sep 24 19:18:29 dist-test-slave-0lgr krb5kdc[1689](info): commencing operation
krb5kdc: starting...
W20260924 19:18:30.066267 11337 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/test-user.keytab.
Sep 24 19:18:30 dist-test-slave-0lgr krb5kdc[1689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:18:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
Entry for principal kudu/127.11.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab.
I20260924 19:18:30.368031 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45115
--webserver_interface=127.11.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45115,127.11.18.125:41803,127.11.18.124:32787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:18:30.717396 1702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:18:30.717789 1702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:30.717921 1702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:30.718058 1702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:18:30.725790 1702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:30.725893 1702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:30.725973 1702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:30.726052 1702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:30.726147 1702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:30.736629 1702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45115,127.11.18.125:41803,127.11.18.124:32787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.126.keytab
--principal=kudu/127.11.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.126:45115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:30.738041 1702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:30.739401 1702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:30.751883 1707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:30.752130 1708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:30.752153 1710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:30.752827 1702 server_base.cc:1034] running on GCE node
I20260924 19:18:30.753777 1702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:30.755609 1702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:30.756898 1702 hybrid_clock.cc:648] HybridClock initialized: now 1790277510756817 us; error 56 us; skew 500 ppm
I20260924 19:18:30.757293 1702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:30 dist-test-slave-0lgr krb5kdc[1689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277510, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:30.761001 1702 init.cc:377] Logged in from keytab as kudu/127.11.18.126@KRBTEST.COM (short username kudu)
I20260924 19:18:30.762704 1702 webserver.cc:460] Webserver started at http://127.11.18.126:37063/ using document root <none> and password file <none>
I20260924 19:18:30.763432 1702 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:30.763525 1702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:30.763795 1702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:30.767048 1702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/data/instance:
uuid: "d9530769628d4810817945ead39de8c2"
format_stamp: "Formatted at 2026-09-24 19:18:30 on dist-test-slave-0lgr"
server_key: "dd058381f12689d67a82039ef84311b5"
server_key_iv: "ada325c686a9b5236223d7fe7202e3f6"
server_key_version: "encryptionkey@0"
I20260924 19:18:30.768074 1702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/wal/instance:
uuid: "d9530769628d4810817945ead39de8c2"
format_stamp: "Formatted at 2026-09-24 19:18:30 on dist-test-slave-0lgr"
server_key: "dd058381f12689d67a82039ef84311b5"
server_key_iv: "ada325c686a9b5236223d7fe7202e3f6"
server_key_version: "encryptionkey@0"
I20260924 19:18:30.773795 1702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 19:18:30.777570 1717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:30.779109 1702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:18:30.779313 1702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/wal
uuid: "d9530769628d4810817945ead39de8c2"
format_stamp: "Formatted at 2026-09-24 19:18:30 on dist-test-slave-0lgr"
server_key: "dd058381f12689d67a82039ef84311b5"
server_key_iv: "ada325c686a9b5236223d7fe7202e3f6"
server_key_version: "encryptionkey@0"
I20260924 19:18:30.779572 1702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:30.798894 1702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:30.803440 1702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:30.803766 1702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:30.823572 1702 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.126:45115
I20260924 19:18:30.823580 1768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.126:45115 every 8 connection(s)
I20260924 19:18:30.824996 1702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/data/info.pb
I20260924 19:18:30.826485 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 1702
I20260924 19:18:30.826671 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-0/wal/instance
I20260924 19:18:30.827234 11337 external_mini_cluster.cc:1428] Setting key f72fa9abdb0ca3fc50a829b4d2693b9f
I20260924 19:18:30.829754 1769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:30.839473 1769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.18.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.125@KRBTEST.COM" created.
Sep 24 19:18:30 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277510, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:18:30.868649 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:30.842743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45115 (local address 127.0.0.1:39674)
0924 19:18:30.843680 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:30.843733 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:30.846608 (+ 2875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:30.847549 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:30.847559 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:30.848369 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:30.848847 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:30.848879 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:30.850570 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:30.850579 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:30.852636 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:30.852656 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:30.853780 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:30.853788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:30.854004 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:30.854689 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:30.854726 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:30.860199 (+ 5473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:30.863776 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:30.863793 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:30.863811 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:30.864489 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:30.864976 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:30.864985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:30.864992 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:30.865226 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:30.866199 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:30.866212 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:30.866548 (+ 336us) client_negotiation.cc:769] Sending connection context
0924 19:18:30.866865 (+ 317us) client_negotiation.cc:240] Negotiation successful
0924 19:18:30.867602 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":83,"thread_start_us":384,"threads_started":1}
I20260924 19:18:30.868698 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:30.842743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.11.18.126:45115)
0924 19:18:30.844651 (+ 1908us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:30.844660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:30.844703 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:18:30.847048 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:30.847054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:30.847300 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:30.847460 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:30.849084 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:30.850348 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:30.852813 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:30.853526 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:30.853685 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:30.860528 (+ 6843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:30.860538 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:30.860551 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:30.860587 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:30.863607 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:30.864647 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:30.864654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:30.864656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:30.864830 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:30.865442 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:30.865446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:30.865448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:30.865993 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:30.866235 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:30.867017 (+ 782us) server_negotiation.cc:299] Negotiation successful
0924 19:18:30.867376 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":1721,"spinlock_wait_cycles":15744,"thread_start_us":1464,"threads_started":1}
I20260924 19:18:30.872021 1769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:18:30.873162 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:30.872742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58730)
0924 19:18:30.872925 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:30.873041 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 19:18:30.873587 1721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
I20260924 19:18:30.875490 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:30.874994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58732)
0924 19:18:30.875190 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:30.875288 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 19:18:30.875816 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:41803: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab.
Entry for principal kudu/127.11.18.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab.
I20260924 19:18:30.893237 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.125:41803
--webserver_interface=127.11.18.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab
--principal=kudu/127.11.18.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45115,127.11.18.125:41803,127.11.18.124:32787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:18:30.925230 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } attempt: 1
I20260924 19:18:30.926597 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:30.926010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58736)
0924 19:18:30.926261 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:30.926431 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:18:30.927688 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:30.927299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58748)
0924 19:18:30.927451 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:30.927536 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 19:18:30.927922 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:41803: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
I20260924 19:18:30.996204 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } attempt: 2
I20260924 19:18:30.997619 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:30.996966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58754)
0924 19:18:30.997264 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:30.997410 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:18:30.998761 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:30.998347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58770)
0924 19:18:30.998518 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:30.998606 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:18:30.999006 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:41803: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
I20260924 19:18:31.090291 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } attempt: 3
I20260924 19:18:31.091571 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.091052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58782)
0924 19:18:31.091287 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.091401 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 19:18:31.092924 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.092460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58792)
0924 19:18:31.092627 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.092722 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 19:18:31.093261 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:41803: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
W20260924 19:18:31.228519 1776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:18:31.228938 1776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:18:31.229067 1776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:31.229207 1776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 19:18:31.236594 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } attempt: 4
W20260924 19:18:31.236922 1776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:31.237043 1776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:31.237185 1776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:31.237286 1776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:31.237402 1776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:31.238086 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.237497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58806)
0924 19:18:31.237830 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.237923 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260924 19:18:31.239259 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.238785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58822)
0924 19:18:31.238970 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.239058 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 19:18:31.239540 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.125:41803: Network error: Client connection negotiation failed: client connection to 127.11.18.125:41803: connect: Connection refused (error 111)
I20260924 19:18:31.248364 1776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45115,127.11.18.125:41803,127.11.18.124:32787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.125.keytab
--principal=kudu/127.11.18.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.125:41803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.18.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:31.249773 1776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:31.251109 1776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:31.263633 1781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:31.263742 1782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:31.263603 1784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:31.264348 1776 server_base.cc:1034] running on GCE node
I20260924 19:18:31.265148 1776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:31.266682 1776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:31.267961 1776 hybrid_clock.cc:648] HybridClock initialized: now 1790277511267900 us; error 52 us; skew 500 ppm
I20260924 19:18:31.268407 1776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:31.300689 1776 init.cc:377] Logged in from keytab as kudu/127.11.18.125@KRBTEST.COM (short username kudu)
I20260924 19:18:31.302521 1776 webserver.cc:460] Webserver started at http://127.11.18.125:33879/ using document root <none> and password file <none>
I20260924 19:18:31.303356 1776 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:31.303544 1776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:31.303923 1776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:31.307262 1776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/data/instance:
uuid: "421935bfaed04386a8f92cc6fd02e792"
format_stamp: "Formatted at 2026-09-24 19:18:31 on dist-test-slave-0lgr"
server_key: "0dce0126cbf8d5aaa08f0e57674961cf"
server_key_iv: "e032a709b7a7d82852bb1760b0974714"
server_key_version: "encryptionkey@0"
I20260924 19:18:31.308125 1776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/wal/instance:
uuid: "421935bfaed04386a8f92cc6fd02e792"
format_stamp: "Formatted at 2026-09-24 19:18:31 on dist-test-slave-0lgr"
server_key: "0dce0126cbf8d5aaa08f0e57674961cf"
server_key_iv: "e032a709b7a7d82852bb1760b0974714"
server_key_version: "encryptionkey@0"
I20260924 19:18:31.313918 1776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:18:31.318109 1791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:31.319692 1776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:18:31.320013 1776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/wal
uuid: "421935bfaed04386a8f92cc6fd02e792"
format_stamp: "Formatted at 2026-09-24 19:18:31 on dist-test-slave-0lgr"
server_key: "0dce0126cbf8d5aaa08f0e57674961cf"
server_key_iv: "e032a709b7a7d82852bb1760b0974714"
server_key_version: "encryptionkey@0"
I20260924 19:18:31.320261 1776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:31.345752 1776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:31.350715 1776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:31.350997 1776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:31.370209 1776 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.125:41803
I20260924 19:18:31.370200 1842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.125:41803 every 8 connection(s)
I20260924 19:18:31.371593 1776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/data/info.pb
I20260924 19:18:31.375365 1843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:31.380599 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 1776
I20260924 19:18:31.380892 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-1/wal/instance
I20260924 19:18:31.381420 11337 external_mini_cluster.cc:1428] Setting key 27e42b0ce1d2ff808aa5247d4d634be5
I20260924 19:18:31.382421 1843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.18.124@KRBTEST.COM; defaulting to no policy
I20260924 19:18:31.409458 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.385456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.11.18.126:45115)
0924 19:18:31.385993 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.385997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.387066 (+ 1069us) server_negotiation.cc:407] Connection header received
0924 19:18:31.389079 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.389085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.389184 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.389306 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.391030 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.391761 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.393364 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.393776 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.393926 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.398654 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.398663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.398670 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.398720 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.403039 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.404425 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.404430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.404432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.404585 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.405309 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.405314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.405316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.406978 (+ 1662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.407279 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.408752 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.408976 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
I20260924 19:18:31.410650 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.386058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45115 (local address 127.0.0.1:39676)
0924 19:18:31.386883 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.386929 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:31.388777 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:31.389615 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:31.389624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:31.390209 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:31.390799 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.390816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.391956 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.391960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.393171 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.393183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.394030 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.394037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.394283 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.395111 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:31.395136 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:31.398321 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:31.403260 (+ 4939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.403273 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.403294 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.404133 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.404731 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.404740 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.404745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.405048 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.407371 (+ 2323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:31.407386 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:31.408201 (+ 815us) client_negotiation.cc:769] Sending connection context
0924 19:18:31.408564 (+ 363us) client_negotiation.cc:240] Negotiation successful
0924 19:18:31.409131 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":353,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.18.124@KRBTEST.COM" created.
I20260924 19:18:31.412822 1843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:18:31.432811 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.413911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58834)
0924 19:18:31.414113 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.414138 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:31.414277 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:31.416528 (+ 2251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:31.416533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:31.417027 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:31.417226 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.417238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.419586 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.419590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.420659 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.420674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.422282 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.422293 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.422501 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.423263 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:31.423295 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:31.426193 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:31.429902 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.429910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.429915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.430313 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.430756 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.430761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.430763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.430922 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.431715 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:31.431719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:31.431862 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 19:18:31.432021 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 19:18:31.432366 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 19:18:31.432947 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.414274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.11.18.125:41803)
0924 19:18:31.415838 (+ 1564us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.415850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.415890 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:18:31.415959 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.415971 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.416303 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.416491 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.417493 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.419402 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.420835 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.422065 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.422217 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.426429 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.426440 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.426455 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.426491 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.429732 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.430464 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.430476 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.430485 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.430608 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.431038 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.431046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.431049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.431486 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.431733 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.432203 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.432527 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":1345,"thread_start_us":1162,"threads_started":1}
I20260924 19:18:31.435799 1843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:18:31.437312 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.436673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53912)
0924 19:18:31.436900 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.437040 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 19:18:31.437717 1793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
I20260924 19:18:31.439453 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.439072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53924)
0924 19:18:31.439248 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.439339 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 19:18:31.439791 1843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.18.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab.
Entry for principal kudu/127.11.18.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab.
I20260924 19:18:31.450784 11337 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
/tmp/dist-test-taskJqTBM_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.124:32787
--webserver_interface=127.11.18.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab
--principal=kudu/127.11.18.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.18.84:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.18.126:45115,127.11.18.125:41803,127.11.18.124:32787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:18:31.489145 1843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 1
I20260924 19:18:31.490463 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.489961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53926)
0924 19:18:31.490166 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.490292 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:18:31.491509 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.491147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53934)
0924 19:18:31.491335 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.491409 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 19:18:31.491755 1843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
I20260924 19:18:31.538849 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } attempt: 5
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277510, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:18:31.554373 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.539742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58846)
0924 19:18:31.540007 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.540034 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:31.540143 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:31.540536 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:31.540540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:31.540996 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:31.541187 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.541196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.542639 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.542643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.543580 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.543592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.544576 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.544581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.544739 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.545476 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:31.545494 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:31.547557 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:31.551627 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.551632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.551635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.552095 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.552561 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.552565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.552568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.552770 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.553455 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:31.553460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:31.553603 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 19:18:31.553825 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 19:18:31.554075 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 19:18:31.554497 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.539915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.11.18.125:41803)
0924 19:18:31.540155 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.540161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.540188 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:18:31.540266 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.540273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.540369 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.540491 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.541383 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.542466 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.543740 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.544388 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.544526 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.547780 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.547790 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.547798 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.547850 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.551429 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.552244 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.552252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.552256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.552434 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.552886 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.552892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.552896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.553260 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.553472 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.553960 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.554203 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:18:31.555744 1769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:18:31.556960 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.556572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53944)
0924 19:18:31.556728 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.556804 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 19:18:31.558264 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.557902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53952)
0924 19:18:31.558074 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.558162 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 19:18:31.558679 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
I20260924 19:18:31.559980 1843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 2
I20260924 19:18:31.560910 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.560517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53962)
0924 19:18:31.560683 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.560767 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:18:31.561905 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.561584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53976)
0924 19:18:31.561724 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.561793 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:18:31.562115 1843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
I20260924 19:18:31.609962 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 1
I20260924 19:18:31.611393 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.610833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53992)
0924 19:18:31.611066 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.611164 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 19:18:31.612706 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.612229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53994)
0924 19:18:31.612446 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.612525 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 19:18:31.613001 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
I20260924 19:18:31.653422 1843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 3
I20260924 19:18:31.654747 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.654259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:53996)
0924 19:18:31.654492 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.654580 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:18:31.655937 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.655511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54010)
0924 19:18:31.655734 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.655806 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260924 19:18:31.656136 1843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
I20260924 19:18:31.681255 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 2
I20260924 19:18:31.682554 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.681999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54014)
0924 19:18:31.682242 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.682328 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:18:31.683501 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.683160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54026)
0924 19:18:31.683313 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.683404 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:18:31.683797 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
I20260924 19:18:31.790108 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 3
I20260924 19:18:31.791640 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.790949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54030)
0924 19:18:31.791251 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.791378 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 19:18:31.792845 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.792382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54034)
0924 19:18:31.792585 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.792686 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 19:18:31.793073 1769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
W20260924 19:18:31.799103 1851 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:18:31.799475 1851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260924 19:18:31.799362 1843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 4
W20260924 19:18:31.799608 1851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:18:31.799747 1851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 19:18:31.800508 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.800054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54042)
0924 19:18:31.800273 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.800360 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:18:31.801520 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.801145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54052)
0924 19:18:31.801294 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.801423 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 19:18:31.801709 1843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.18.124:32787: Network error: Client connection negotiation failed: client connection to 127.11.18.124:32787: connect: Connection refused (error 111)
W20260924 19:18:31.807698 1851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:18:31.807824 1851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:18:31.807940 1851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:18:31.808022 1851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:18:31.808126 1851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:18:31.818578 1851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.18.84:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.18.126:45115,127.11.18.125:41803,127.11.18.124:32787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/krb5kdc/kudu_127.11.18.124.keytab
--principal=kudu/127.11.18.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.18.124:32787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.18.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:18:31.819964 1851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:18:31.821477 1851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:18:31.835747 1856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:31.835791 1857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:18:31.836092 1859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:18:31.837237 1851 server_base.cc:1034] running on GCE node
I20260924 19:18:31.838246 1851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:18:31.839836 1851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:18:31.841120 1851 hybrid_clock.cc:648] HybridClock initialized: now 1790277511841067 us; error 60 us; skew 500 ppm
I20260924 19:18:31.841567 1851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:18:31.845212 1851 init.cc:377] Logged in from keytab as kudu/127.11.18.124@KRBTEST.COM (short username kudu)
I20260924 19:18:31.846989 1851 webserver.cc:460] Webserver started at http://127.11.18.124:46843/ using document root <none> and password file <none>
I20260924 19:18:31.847774 1851 fs_manager.cc:359] Metadata directory not provided
I20260924 19:18:31.847919 1851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:18:31.848261 1851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:18:31.851560 1851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/data/instance:
uuid: "02da82f04e49487aabd2999e3e5bc597"
format_stamp: "Formatted at 2026-09-24 19:18:31 on dist-test-slave-0lgr"
server_key: "686f69885a0725dce3a8c4fa0b61cfa4"
server_key_iv: "65e34dabc41bf89727b2a70823487482"
server_key_version: "encryptionkey@0"
I20260924 19:18:31.852504 1851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/wal/instance:
uuid: "02da82f04e49487aabd2999e3e5bc597"
format_stamp: "Formatted at 2026-09-24 19:18:31 on dist-test-slave-0lgr"
server_key: "686f69885a0725dce3a8c4fa0b61cfa4"
server_key_iv: "65e34dabc41bf89727b2a70823487482"
server_key_version: "encryptionkey@0"
I20260924 19:18:31.858528 1851 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 19:18:31.862629 1866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:18:31.864182 1851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260924 19:18:31.864398 1851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/wal
uuid: "02da82f04e49487aabd2999e3e5bc597"
format_stamp: "Formatted at 2026-09-24 19:18:31 on dist-test-slave-0lgr"
server_key: "686f69885a0725dce3a8c4fa0b61cfa4"
server_key_iv: "65e34dabc41bf89727b2a70823487482"
server_key_version: "encryptionkey@0"
I20260924 19:18:31.864651 1851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:18:31.882813 1851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:18:31.887853 1851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:18:31.888156 1851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:18:31.908406 1851 rpc_server.cc:307] RPC server started. Bound to: 127.11.18.124:32787
I20260924 19:18:31.908442 1917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.18.124:32787 every 8 connection(s)
I20260924 19:18:31.909866 1851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/data/info.pb
I20260924 19:18:31.913991 1918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:18:31.918406 11337 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu as pid 1851
I20260924 19:18:31.918608 11337 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqTBM_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277128821086-11337-0/minicluster-data/master-2/wal/instance
I20260924 19:18:31.919081 11337 external_mini_cluster.cc:1428] Setting key 424543a2702d0ff6c982eed0214be58e
I20260924 19:18:31.922103 1918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:18:31.942744 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.926353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.11.18.126:45115)
0924 19:18:31.926935 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.926940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.927425 (+ 485us) server_negotiation.cc:407] Connection header received
0924 19:18:31.927645 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.927650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.927728 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.927827 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.929023 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.929966 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.931692 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.932366 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.932512 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.936467 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.936474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.936480 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.936531 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.939559 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.940445 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.940449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.940451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.940575 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.941034 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.941038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.941040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.941509 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.941669 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.942297 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.942465 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":252,"threads_started":1}
I20260924 19:18:31.943974 1923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.926373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.126:45115 (local address 127.0.0.1:39684)
0924 19:18:31.927265 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.927322 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:31.927479 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:31.927910 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:31.927917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:31.928600 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:31.928847 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.928856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.930127 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.930132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.931527 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.931539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.932626 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.932635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.932833 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.933486 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:31.933509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:31.936162 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:31.939689 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.939699 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.939712 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.940335 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.940686 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.940690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.940694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.940850 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.941652 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:31.941662 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:31.941892 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 19:18:31.942077 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 19:18:31.942555 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":137,"spinlock_wait_cycles":40448,"thread_start_us":474,"threads_started":1}
I20260924 19:18:31.943996 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.920939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.11.18.124:32787)
0924 19:18:31.921835 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.921846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.921886 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:18:31.923436 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.923468 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.923974 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.924430 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.926005 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.927960 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.929606 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.930533 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.930736 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.935525 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.935536 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.935553 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.935598 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.938770 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.939689 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.939699 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.939709 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.939936 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.940344 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.940349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.940352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.941116 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.941421 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.941892 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.942455 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":676,"spinlock_wait_cycles":384,"thread_start_us":466,"threads_started":1}
I20260924 19:18:31.945758 1918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.126@KRBTEST.COM
I20260924 19:18:31.960667 1923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.946688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.125:41803 (local address 127.0.0.1:58848)
0924 19:18:31.946964 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.946980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:31.947119 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:31.947708 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:31.947711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:31.948174 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:31.948353 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.948362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.949385 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.949391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.950274 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.950285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.951014 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.951020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.951223 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.951816 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:31.951834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:31.954226 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:31.957984 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.957989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.957993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.958568 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.958971 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.958976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.958980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.959135 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.959679 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:31.959683 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:31.959841 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 19:18:31.960008 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 19:18:31.960299 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260924 19:18:31.960692 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.946643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.11.18.125:41803)
0924 19:18:31.946880 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.946886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.947049 (+ 163us) server_negotiation.cc:407] Connection header received
0924 19:18:31.947349 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.947354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.947439 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.947531 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.948501 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.949211 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.950420 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.950859 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.950991 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.954492 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.954498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.954501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.954535 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.957827 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.958695 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.958701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.958705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.958850 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.959246 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.959253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.959256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.959550 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.959704 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.960097 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.960317 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 19:18:31.961815 1918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:18:31.962419 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.947489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.11.18.126:45115)
0924 19:18:31.947909 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.947913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.947926 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:18:31.947976 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.947980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.948052 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.948243 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.949526 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.950209 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.951482 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.951909 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.952048 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.956786 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.956791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.956794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.956829 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.959542 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.960272 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.960279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.960347 (+ 68us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.960470 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.960927 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.960934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.960939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.961263 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.961596 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.961896 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.962138 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.124@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
I20260924 19:18:31.970306 1769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 4
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.18.125@KRBTEST.COM
I20260924 19:18:31.978055 1923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.962671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54070)
0924 19:18:31.962849 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.962878 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:31.962982 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:31.963273 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:31.963280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:31.963702 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:31.963911 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.963925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.965538 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.965542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.966426 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.966440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.967129 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.967134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.967345 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.967921 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:31.967938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:31.970538 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:31.975618 (+ 5080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.975623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.975627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.975973 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.976345 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.976351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.976354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.976488 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.977067 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:31.977072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:31.977216 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 19:18:31.977405 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 19:18:31.977793 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":20}
Sep 24 19:18:31 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277510, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.126@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
I20260924 19:18:31.979660 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.962857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.11.18.124:32787)
0924 19:18:31.963007 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.963013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.963032 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:18:31.963087 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.963092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.963146 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.963221 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.964266 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.965408 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.966572 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.966991 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.967126 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.970753 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.970758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.970761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.970790 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.975485 (+ 4695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.976092 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.976098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.976102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.976229 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.976596 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.976604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.976607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.976879 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.977080 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.977493 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.979296 (+ 1803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1532,"server-negotiator.queue_time_us":51}
I20260924 19:18:31.980515 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.964117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.11.18.125:41803)
0924 19:18:31.964284 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.964288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.964303 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:18:31.964531 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.964535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.964585 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.964657 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.966073 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.966764 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.967933 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.968376 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.968507 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.973391 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.973401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.973404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.973436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.976821 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.978330 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.978337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.978341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.978513 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.978952 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.978956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.978958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.979215 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.979510 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.979928 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.980179 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:18:31.986243 11337 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:18:31.986261 1918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597: Bootstrap starting.
I20260924 19:18:31.987799 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.971067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54076)
0924 19:18:31.971276 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:31.971300 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:31.971444 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:31.972169 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:31.972175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:31.972665 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:31.972814 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.972823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.974441 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.974447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.975949 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:31.975965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.977011 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.977017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:31.977218 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.977830 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:31.977852 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:31.980350 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:31.984656 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.984662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.984666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.985077 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.985524 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:31.985529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:31.985532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:31.986184 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:31.986839 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:31.986846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:31.987019 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 19:18:31.987171 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 19:18:31.987451 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:18:31.988014 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:31.971094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.11.18.124:32787)
0924 19:18:31.971796 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:31.971805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:31.971830 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:18:31.971884 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:31.971892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:31.971970 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:31.972122 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:31.972964 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.974229 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.976128 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:31.976837 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:31.976960 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:31.980772 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:31.980782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:31.980788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:31.980823 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:31.984525 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.985192 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.985199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.985203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.985402 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:31.986313 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:31.986319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:31.986323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:31.986690 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:31.986817 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:31.987274 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 19:18:31.987582 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":306,"threads_started":1}
I20260924 19:18:31.990360 1918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:31.991696 1918 log.cc:824] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:31.989109 1744 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d9530769628d4810817945ead39de8c2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39700
I20260924 19:18:31.994535 1769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2: Bootstrap starting.
I20260924 19:18:31.995170 1918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597: No bootstrap required, opened a new log
I20260924 19:18:31.998495 1769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:31.999794 1769 log.cc:824] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:32.003216 1769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2: No bootstrap required, opened a new log
I20260924 19:18:32.005635 1918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } }
I20260924 19:18:32.006072 1918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02da82f04e49487aabd2999e3e5bc597, State: Initialized, Role: FOLLOWER
I20260924 19:18:32.006579 1918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } }
I20260924 19:18:32.008666 1927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.009068 1927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:32.009995 1918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:32.011193 1769 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } }
I20260924 19:18:32.011672 1769 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9530769628d4810817945ead39de8c2, State: Initialized, Role: FOLLOWER
I20260924 19:18:32.012261 1769 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } }
I20260924 19:18:32.013273 1933 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.013727 1933 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:32.014806 1769 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:32.014876 1744 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:18:32.015095 1744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:32.017964 1744 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } }
I20260924 19:18:32.020139 1893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9530769628d4810817945ead39de8c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "02da82f04e49487aabd2999e3e5bc597"
I20260924 19:18:32.020452 1744 leader_election.cc:288] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [CANDIDATE]: Term 1 election: Requested vote from peers 421935bfaed04386a8f92cc6fd02e792 (127.11.18.125:41803), 02da82f04e49487aabd2999e3e5bc597 (127.11.18.124:32787)
I20260924 19:18:32.020717 1893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:32.026605 1893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9530769628d4810817945ead39de8c2 in term 1.
I20260924 19:18:32.027894 1719 leader_election.cc:302] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02da82f04e49487aabd2999e3e5bc597, d9530769628d4810817945ead39de8c2; no voters:
I20260924 19:18:32.029134 1933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:18:32.038498 1933 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [term 1 LEADER]: Becoming Leader. State: Replica: d9530769628d4810817945ead39de8c2, State: Running, Role: LEADER
I20260924 19:18:32.040180 1933 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } }
I20260924 19:18:32.048708 1949 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 19:18:32.056905 1723 tablet.cc:2369] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:18:32.052205 1851 master_runner.cc:186] Error getting master registration for 127.11.18.126:45115: OK, instance_id { permanent_uuid: "d9530769628d4810817945ead39de8c2" instance_seqno: 1790277510817834 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:18:32.065141 1851 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:18:32.068147 1944 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9530769628d4810817945ead39de8c2. Latest consensus state: current_term: 1 leader_uuid: "d9530769628d4810817945ead39de8c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.068645 1944 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:32.087167 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.061591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.11.18.124:32787)
0924 19:18:32.061829 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.061835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.067990 (+ 6155us) server_negotiation.cc:407] Connection header received
0924 19:18:32.068517 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.068522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.068619 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.068721 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.071490 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.072652 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.074550 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.075190 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.075298 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.079116 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:32.079126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:32.079131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:32.079176 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:32.082935 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:32.083928 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:32.083934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:32.083937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:32.084111 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:32.084613 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:32.084618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:32.084621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:32.085041 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:32.085289 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.086055 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.086532 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260924 19:18:32.075433 1702 master_runner.cc:186] Error getting master registration for 127.11.18.125:41803: OK, instance_id { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" instance_seqno: 1790277511364860 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:18:32.088500 1702 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:18:32.092026 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.055028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.11.18.126:45115)
0924 19:18:32.055239 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.055244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.067399 (+ 12155us) server_negotiation.cc:407] Connection header received
0924 19:18:32.068085 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.068093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.068198 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.068323 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.072857 (+ 4534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.074093 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.076334 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.077102 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.077233 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.081170 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:32.081179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:32.081188 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:32.081238 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:32.085192 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:32.086117 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:32.086121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:32.086125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:32.086281 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:32.086731 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:32.086737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:32.086740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:32.087132 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:32.090588 (+ 3456us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.091217 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.091641 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:18:32.094512 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.065985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.11.18.125:41803)
0924 19:18:32.066187 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.066194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.069728 (+ 3534us) server_negotiation.cc:407] Connection header received
0924 19:18:32.069904 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.069909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.069990 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.070098 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.078054 (+ 7956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.079168 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.082279 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.082973 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.083075 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.086595 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:32.086602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:32.086606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:32.086649 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:32.090330 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:32.091402 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:32.091411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:32.091417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:32.091625 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:32.092678 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:32.092683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:32.092686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:32.093061 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:32.093345 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.093896 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.094140 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 19:18:32.096899 1960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:18:32.097007 1960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:18:32.097196 1959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:18:32.099435 1959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:18:32.100930 1843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } attempt: 5
I20260924 19:18:32.110239 1893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [term 1 FOLLOWER]: Refusing update from remote peer d9530769628d4810817945ead39de8c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:18:32.111402 1933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 24 19:18:32 dist-test-slave-0lgr krb5kdc[1689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.18.125@KRBTEST.COM for kudu/127.11.18.124@KRBTEST.COM
I20260924 19:18:32.126294 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.101919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.11.18.124:32787)
0924 19:18:32.102100 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.102108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.102129 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:18:32.102196 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.102204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.102286 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.102397 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.103328 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.104623 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.107194 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.107860 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.108003 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.114772 (+ 6769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:18:32.114781 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:18:32.114788 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:18:32.114833 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:18:32.119290 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:32.123662 (+ 4372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:32.123670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:32.123676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:32.123898 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:18:32.124677 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:18:32.124685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:18:32.124691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:18:32.125194 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:18:32.125461 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.125841 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.126050 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:18:32.126451 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.101571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.18.124:32787 (local address 127.0.0.1:54098)
0924 19:18:32.101761 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:18:32.101783 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:18:32.101901 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:18:32.102438 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:18:32.102444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:18:32.102958 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:18:32.103157 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:32.103168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.104814 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.104821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:32.106236 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:18:32.106253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.108026 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.108031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:18:32.108272 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.108987 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:18:32.109013 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:18:32.114525 (+ 5512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:18:32.119562 (+ 5037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:32.119573 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:32.119580 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:32.120354 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:32.124220 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:18:32.124228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:18:32.124235 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:18:32.124482 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:18:32.125354 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:18:32.125360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:18:32.125535 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 19:18:32.125705 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 19:18:32.126047 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:18:32.127446 1927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9530769628d4810817945ead39de8c2. Latest consensus state: current_term: 1 leader_uuid: "d9530769628d4810817945ead39de8c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.127758 1927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:32.133447 1843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792: Bootstrap starting.
I20260924 19:18:32.133440 1962 mvcc.cc:205] Tried to move back new op lower bound from 7332976689591578624 to 7332976689335754752. Current Snapshot: MvccSnapshot[applied={T|T < 7332976689591578624}]
I20260924 19:18:32.138641 1843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792: Neither blocks nor log segments found. Creating new log.
I20260924 19:18:32.140534 1843 log.cc:824] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792: Log is configured to *not* fsync() on all Append() calls
I20260924 19:18:32.144759 1927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9530769628d4810817945ead39de8c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.145036 1927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:32.146950 1944 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9530769628d4810817945ead39de8c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.147280 1944 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:32.149262 1959 catalog_manager.cc:1347] Generated new cluster ID: a73aeb20ffeb48fb80bae9a30f022ba5
I20260924 19:18:32.149394 1959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:18:32.149596 1843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792: No bootstrap required, opened a new log
I20260924 19:18:32.158679 1843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } }
I20260924 19:18:32.159567 1843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421935bfaed04386a8f92cc6fd02e792, State: Initialized, Role: FOLLOWER
I20260924 19:18:32.160137 1843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } }
I20260924 19:18:32.161844 1968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.162719 1843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:18:32.163215 1968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:32.163302 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9530769628d4810817945ead39de8c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "421935bfaed04386a8f92cc6fd02e792"
I20260924 19:18:32.163331 1817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:18:32.170744 1817 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [term 1 FOLLOWER]: Refusing update from remote peer d9530769628d4810817945ead39de8c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:18:32.174556 1818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9530769628d4810817945ead39de8c2 in term 1.
I20260924 19:18:32.174701 1959 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:18:32.176676 1959 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:18:32.177119 1933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:18:32.201939 1959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2: Generated new TSK 0
I20260924 19:18:32.203172 1959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:18:32.217291 1968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9530769628d4810817945ead39de8c2. Latest consensus state: current_term: 1 leader_uuid: "d9530769628d4810817945ead39de8c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.217772 1968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:32.217926 1933 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9530769628d4810817945ead39de8c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.217964 1973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9530769628d4810817945ead39de8c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9530769628d4810817945ead39de8c2" member_type: VOTER last_known_addr { host: "127.11.18.126" port: 45115 } } peers { permanent_uuid: "421935bfaed04386a8f92cc6fd02e792" member_type: VOTER last_known_addr { host: "127.11.18.125" port: 41803 } } peers { permanent_uuid: "02da82f04e49487aabd2999e3e5bc597" member_type: VOTER last_known_addr { host: "127.11.18.124" port: 32787 } } }
I20260924 19:18:32.218272 1933 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9530769628d4810817945ead39de8c2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:18:32.218320 1973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:18:32.250404 1984 catalog_manager.cc:1259] Loaded cluster ID: a73aeb20ffeb48fb80bae9a30f022ba5
I20260924 19:18:32.251008 1984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792: loading cluster ID for follower catalog manager: success
I20260924 19:18:32.255106 1984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792: acquiring CA information for follower catalog manager: success
I20260924 19:18:32.257861 1984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 421935bfaed04386a8f92cc6fd02e792: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:18:32.300552 1922 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.290997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.11.18.126:45115)
0924 19:18:32.291238 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.291245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.298065 (+ 6820us) server_negotiation.cc:407] Connection header received
0924 19:18:32.298398 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.298409 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.298522 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.298623 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.300207 (+ 1584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39712: BlockingRecv error: recv got EOF from 127.0.0.1:39712 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 19:18:32.306216 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.291111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.11.18.124:32787)
0924 19:18:32.291403 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.291410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.304000 (+ 12590us) server_negotiation.cc:407] Connection header received
0924 19:18:32.304493 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.304500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.304593 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.304666 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.305945 (+ 1279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54114: BlockingRecv error: recv got EOF from 127.0.0.1:54114 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
W20260924 19:18:32.306682 1848 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.297079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.11.18.125:41803)
0924 19:18:32.297385 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.297393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.304760 (+ 7367us) server_negotiation.cc:407] Connection header received
0924 19:18:32.305190 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.305195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.305275 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.305396 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.306466 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58878: BlockingRecv error: recv got EOF from 127.0.0.1:58878 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 19:18:32.321432 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.312247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.11.18.125:41803)
0924 19:18:32.312562 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.312568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.313899 (+ 1331us) server_negotiation.cc:407] Connection header received
0924 19:18:32.314235 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.314242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.314462 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.314645 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.315170 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.316413 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.318523 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.319254 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.319434 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.319917 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.320653 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.320817 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.320947 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.321054 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 19:18:32.331676 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.329386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.11.18.124:32787)
0924 19:18:32.329586 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.329592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.330109 (+ 517us) server_negotiation.cc:407] Connection header received
0924 19:18:32.330332 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.330339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.330468 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.330578 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.331437 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54116: BlockingRecv error: recv got EOF from 127.0.0.1:54116 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:18:32.344162 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.336566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.11.18.125:41803)
0924 19:18:32.336770 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.336776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.337282 (+ 506us) server_negotiation.cc:407] Connection header received
0924 19:18:32.337612 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.337619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.337705 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.337765 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.338419 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.339584 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.341790 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.342529 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.342634 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.343237 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.343415 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.343480 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.343728 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.343826 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 19:18:32.355543 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.352299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.11.18.124:32787)
0924 19:18:32.352486 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.352492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.353944 (+ 1452us) server_negotiation.cc:407] Connection header received
0924 19:18:32.354135 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.354141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.354244 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.354358 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.355284 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54120: BlockingRecv error: recv got EOF from 127.0.0.1:54120 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:18:32.371287 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.363784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.11.18.125:41803)
0924 19:18:32.364052 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.364058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.364624 (+ 566us) server_negotiation.cc:407] Connection header received
0924 19:18:32.364848 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.364855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.364976 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.365079 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.365586 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.366826 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.368933 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.369734 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.369885 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.370388 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.370568 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.370634 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.370868 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.370956 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260924 19:18:32.414021 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.411575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.11.18.124:32787)
0924 19:18:32.411786 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.411796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.412415 (+ 619us) server_negotiation.cc:407] Connection header received
0924 19:18:32.412660 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.412666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.412791 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.412929 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.413764 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54124: BlockingRecv error: recv got EOF from 127.0.0.1:54124 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:18:32.427551 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.421495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.11.18.125:41803)
0924 19:18:32.421724 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.421728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.422318 (+ 590us) server_negotiation.cc:407] Connection header received
0924 19:18:32.422529 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.422541 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.422612 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.422729 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.423148 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.424060 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.425804 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.426278 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.426438 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.426842 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.426948 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.427049 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.427192 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.427272 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 19:18:32.439357 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.434454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.11.18.124:32787)
0924 19:18:32.434825 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.434834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.437927 (+ 3093us) server_negotiation.cc:407] Connection header received
0924 19:18:32.438139 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.438145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.438235 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.438348 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.439100 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54140: BlockingRecv error: recv got EOF from 127.0.0.1:54140 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
I20260924 19:18:32.461186 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.455351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.11.18.125:41803)
0924 19:18:32.455597 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.455606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.456143 (+ 537us) server_negotiation.cc:407] Connection header received
0924 19:18:32.456369 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.456375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.456457 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.456559 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.456959 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.457813 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.459151 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.459826 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.459969 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.460357 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.460504 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.460603 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.460769 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.460851 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 19:18:32.471557 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.467378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.11.18.124:32787)
0924 19:18:32.467604 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.467614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.470058 (+ 2444us) server_negotiation.cc:407] Connection header received
0924 19:18:32.470276 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.470283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.470375 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.470479 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.471273 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54154: BlockingRecv error: recv got EOF from 127.0.0.1:54154 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:18:32.496842 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.491137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.11.18.125:41803)
0924 19:18:32.491370 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.491379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.491893 (+ 514us) server_negotiation.cc:407] Connection header received
0924 19:18:32.492129 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.492133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.492201 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.492308 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.492723 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.493666 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.495077 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.495509 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.495646 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.496073 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.496204 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.496318 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.496457 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.496548 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 19:18:32.504796 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.502991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.11.18.124:32787)
0924 19:18:32.503200 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.503207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.503441 (+ 234us) server_negotiation.cc:407] Connection header received
0924 19:18:32.503648 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.503654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.503736 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.503845 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.504583 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54170: BlockingRecv error: recv got EOF from 127.0.0.1:54170 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:18:32.546651 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.540350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.11.18.125:41803)
0924 19:18:32.540609 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.540619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.541231 (+ 612us) server_negotiation.cc:407] Connection header received
0924 19:18:32.541495 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.541501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.541592 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.541701 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.542149 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.542877 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.544545 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.545098 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.545247 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.545784 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.545898 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.546002 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.546155 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.546246 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 19:18:32.557911 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.555521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.11.18.124:32787)
0924 19:18:32.555766 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.555776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.556375 (+ 599us) server_negotiation.cc:407] Connection header received
0924 19:18:32.556597 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.556609 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.556715 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.556842 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.557658 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54174: BlockingRecv error: recv got EOF from 127.0.0.1:54174 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:18:32.634253 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.626903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.11.18.125:41803)
0924 19:18:32.627102 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.627107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.627624 (+ 517us) server_negotiation.cc:407] Connection header received
0924 19:18:32.627829 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.627836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.627935 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.628062 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.628518 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.629746 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.631771 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.632549 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.632695 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.633176 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.633375 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.633470 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.633638 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.633725 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 19:18:32.644411 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.641984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.11.18.124:32787)
0924 19:18:32.642183 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.642189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.642763 (+ 574us) server_negotiation.cc:407] Connection header received
0924 19:18:32.642978 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.642986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.643088 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.643203 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.644174 (+ 971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54176: BlockingRecv error: recv got EOF from 127.0.0.1:54176 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:18:32.784283 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:32.778786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.11.18.125:41803)
0924 19:18:32.779028 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.779034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.779336 (+ 302us) server_negotiation.cc:407] Connection header received
0924 19:18:32.779531 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.779536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.779618 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.779699 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:32.780093 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.780835 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.782398 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:32.782830 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:32.782930 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:32.783560 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:32.783678 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:32.783749 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:32.783894 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 19:18:32.783963 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 19:18:32.793551 1924 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:18:32.791728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.11.18.124:32787)
0924 19:18:32.791967 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:32.791972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:32.792177 (+ 205us) server_negotiation.cc:407] Connection header received
0924 19:18:32.792327 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:32.792331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:32.792411 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:32.792506 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:18:32.793176 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54182: BlockingRecv error: recv got EOF from 127.0.0.1:54182 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:18:33.054859 1949 catalog_manager.cc:1259] Loaded cluster ID: a73aeb20ffeb48fb80bae9a30f022ba5
I20260924 19:18:33.055047 1949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597: loading cluster ID for follower catalog manager: success
I20260924 19:18:33.060290 1949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597: acquiring CA information for follower catalog manager: success
I20260924 19:18:33.063088 1949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02da82f04e49487aabd2999e3e5bc597: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:18:33.067988 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:33.061682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.11.18.125:41803)
0924 19:18:33.061889 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:33.061896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:33.062321 (+ 425us) server_negotiation.cc:407] Connection header received
0924 19:18:33.062546 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:33.062551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:33.062648 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:33.062752 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:33.063262 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.064064 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.066177 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.066635 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.066743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:33.067237 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:33.067395 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:33.067470 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:33.067640 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 19:18:33.067708 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:18:33.083140 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:33.075547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.11.18.124:32787)
0924 19:18:33.075756 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:33.075779 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:33.076000 (+ 221us) server_negotiation.cc:407] Connection header received
0924 19:18:33.076241 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:33.076245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:33.076459 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:33.076586 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:33.077124 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.078192 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.080306 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.080863 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.080981 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:33.081658 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:33.082316 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:33.082447 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:33.082591 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:18:33.082669 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:18:33.112031 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:33.092171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.11.18.126:45115)
0924 19:18:33.102817 (+ 10646us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:33.102824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:33.103388 (+ 564us) server_negotiation.cc:407] Connection header received
0924 19:18:33.103673 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:33.103682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:33.103939 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:33.104852 (+ 913us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:33.105045 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.106204 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.108453 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.109050 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.109758 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:33.109977 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:33.110565 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:33.111443 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:33.111503 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:18:33.111613 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10477,"thread_start_us":10330,"threads_started":1}
I20260924 19:18:33.112385 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:33.091759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.11.18.125:41803)
0924 19:18:33.091965 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:33.091972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:33.104777 (+ 12805us) server_negotiation.cc:407] Connection header received
0924 19:18:33.104962 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:33.104969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:33.105071 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:33.105152 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:33.106697 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.107796 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.110136 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.110814 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.110912 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:33.111592 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:33.111715 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:33.111789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:33.112004 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 19:18:33.112088 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:18:33.117972 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1702
I20260924 19:18:33.119213 1764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:33.123962 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:18:33.092117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.11.18.124:32787)
0924 19:18:33.092289 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:18:33.092294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:18:33.104163 (+ 11869us) server_negotiation.cc:407] Connection header received
0924 19:18:33.105168 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:18:33.105175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:18:33.105263 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:18:33.105353 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:18:33.106724 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.107868 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.119594 (+ 11726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:18:33.120697 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:18:33.120864 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:18:33.122201 (+ 1337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:18:33.122362 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:18:33.122458 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:18:33.123459 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0924 19:18:33.123554 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:18:33.241475 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 1702
W20260924 19:18:33.270474 1794 connection.cc:535] server connection from 127.0.0.1:58846 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:18:33.271433 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1776
I20260924 19:18:33.272516 1838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:33.386312 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 1776
I20260924 19:18:33.408248 11337 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu pid 1851
I20260924 19:18:33.409533 1913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:18:33.530258 11337 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqTBM_/build/asan/bin/kudu with pid 1851
2026-09-24T19:18:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4637 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9151 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (384673 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST